Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Show a partial PMG read as degraded, not unreachable

rcourtman53990d55f025
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 552c227134..ad929610dc 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -9973,6 +9973,21 @@ than by a non-default org's detached config copy. Agent connection rows keep
their own heartbeat cutoff. Agent registration, enrolment, install, update,
removal and report identity are unchanged.
+### Platform partial reads show as degraded rows
+
+`frontend-modern/src/components/Settings/useConnectionsLedger.ts`,
+`connectionsTableModel.ts` and `InfrastructureSourceManager.tsx` changed only
+so a platform connection in the new `degraded` state (a poll that answered but
+left one reading unread, #2960) renders an amber "Degraded" badge, counts as a
+contributing source, and shows its error banner in warning tone naming the
+failed reading ("Queue status could not be read: ...") through
+`formatPartialReadErrorMessage`. Every other error banner keeps the critical
+treatment. Agent connections never take the `degraded` state, so
+`agentAttachmentProblem`, agent registration, enrolment, install, update,
+removal and report identity are unchanged. The rendered rows are pinned in
+`InfrastructureWorkspace.test.tsx` ("shows a partial read as a degraded
+source..." and "keeps an outage error banner critical").
+
### Host snapshots report a linked agent's own heartbeat
`internal/monitoring/monitor.go` changed only so a host produced from the read
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 93db1538ba..3c56bc1cd4 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -253,6 +253,20 @@ and exactly-once callbacks with shadow parity. `TestInterruptDiscreteRun` in
acknowledgement, no fabricated resolution and idempotent interruption. These
synthetic source checks do not establish native outage or destination delivery.
+### Partial platform reads alert as degraded warnings
+
+A platform connection whose poll answered but left one reading unread (poll
+error category `partial`) reaches `CheckConnection` as state `degraded`. It is
+a degraded observation of the same `connection-degraded` discrete state, at
+warning severity, never critical: it activates after the usual three
+observations, keeps an open outage open (a missing reading is not recovery
+evidence), and moves an already-firing critical outage down to warning once the
+connection answers again. Only `active` runs the recovery confirmation. The
+alert message and metadata keep the failed read (`stateReason`, `lastError`,
+`lastErrorCategory: partial`). No new alert family, threshold, policy or route
+is added. `TestConnectionDegradedPartialReadAlertsAtWarning` in
+`operational_contract_test.go` pins this.
+
### Metric timing requires continuous observed evidence
Unknown metric evidence interrupts timing, not the occurrence. Missing or
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 6e62140042..74581bf142 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -12586,6 +12586,29 @@ finding's resource reference through `Monitor.GetUnifiedStructureReadState`; in
mock mode that is the view built for the current fixture structure, outside it
the read state it used before.
+### A partial platform read is degraded, not unreachable (#2960)
+
+`deriveConnectionState` (`internal/api/connections_aggregator.go`) maps an
+outstanding poll error with category `monitoring.PollErrorCategoryPartial` to
+the connection state `degraded`, with the failed read as `stateReason` and
+`lastError` (whose `category` stays `partial` on the wire and in the
+`frontend-modern/src/api/connections.ts` `ConnectionError` type). The poll
+reached the connection, authenticated and read its version, so it is not
+`unreachable`, and a 401/403 text from one optional endpoint does not make the
+credential `unauthorized`. A partial connection whose last success has also
+aged past the stale cutoff is still `stale`. `degraded` rolls up as fleet
+adapter health `degraded`, keeps credential status `verified`, does not block
+rollout or runtime workload inventory, and ranks between `pending` and `stale`
+in the system-member and ledger severity orders. The Connections page shows it
+as an amber "Degraded" badge with a "Degraded" problem line and the failed
+read, and counts it as a contributing source. No route, request field or
+status code changes; `degraded` is a new value of the existing `state` field.
+`TestDeriveConnectionState_PartialReadIsDegraded`,
+`TestDeriveConnectionState_PartialReadStillGoesStale` and
+`TestBuildConnections_PMGPartialReadIsDegradedNotBlocked` in
+`connections_aggregator_test.go`, and the degraded-row case in
+`useConnectionsLedger.test.ts`, pin this.
+
### Loaded availability Settings honours evidence expiry (10 October 2026)
The existing availability Settings client passes the shared current time to
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index b5515d262c..7848744454 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -10066,6 +10066,17 @@ transcript block moved to `frontend-modern/src/components/patrol/` with its
at `/settings/pulse-intelligence/agents`; the old `/assistant` path redirects
there.
+### Source-row error banners follow the row's severity
+
+`InfrastructureSourceManager.tsx` renders a row's error banner, in both the
+table and the narrow card layout, from one `rowErrorBannerClass` helper keyed
+on `row.lastErrorTone`. `useConnectionsLedger.ts` sets that tone to `warning`
+for a partial-read error on a `degraded` connection and to `critical` for every
+other error, so a reachable source with one failed reading gets an amber
+banner beside its amber "Degraded" badge, while outages keep the rose banner.
+No new component, primitive or badge vocabulary is added. The rendered tones are
+pinned in `InfrastructureWorkspace.test.tsx`.
+
### Configuration transfers retain their confirmed workspace
Recovery's overview identifies the selected-workspace configuration export as
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 4e0cf6f55b..d0e779f1de 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -857,6 +857,11 @@ datastore and node-status reads out of the scheduler outcome.
`TestRecordTaskOutcomeReadErrorSchedulesAsSuccess` in
`monitor_additional_test.go` pins the scheduler bookkeeping, which records the
success and the outstanding read error in one update.
+`poll_error_category.go` exports the category as `PollErrorCategoryPartial`,
+the one value the connections ledger reads to present such a poll as a
+degraded, still-reachable connection (see the API contract's partial-read
+clause) rather than an outage; `TestRecordTaskOutcomeReadErrorSchedulesAsSuccess`
+pins that the recorded category is that exported value.
The PMG client rejects omitted/null object payloads for version, mail, queue
and quarantine instead of materialising them as zero structs. Explicit
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 979511f1b2..5637614157 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -2811,7 +2811,7 @@ recovery scope, or a storage/recovery-owned secret source.
`internal/api/configapi/config_handlers.go` and `internal/api/configapi/config_node_handlers.go`
may carry the new per-instance `Enabled`/`Disabled` round-trip, but
storage and recovery consumers must not reinterpret the derived
- connection `state` (active/paused/unauthorized/unreachable/stale/pending)
+ connection `state` (active/paused/unauthorized/unreachable/stale/pending/degraded)
as storage health, backup-job posture, or recovery verification state;
they also must not repurpose the shared probe route as a generic internal
reachability scanner. Metadata, link-local, multicast, and unspecified
diff --git a/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.json b/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.json
new file mode 100644
index 0000000000..851f1feb1f
--- /dev/null
+++ b/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.json
@@ -0,0 +1,47 @@
+{
+ "version": 1,
+ "base_sha": "0d7f2b7997dad75977d84ae35590f41d10374bd4",
+ "verified_at": "2026-10-26T11:50:00Z",
+ "result": "passed",
+ "changed_paths": [
+ "frontend-modern/src/api/connections.ts",
+ "frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx",
+ "frontend-modern/src/components/Settings/connectionsTableModel.ts",
+ "frontend-modern/src/components/Settings/useConnectionsLedger.ts",
+ "frontend-modern/src/utils/connectionErrorPresentation.ts"
+ ],
+ "content_sha256": {
+ "frontend-modern/src/api/connections.ts": "ca3393144dc7949b40dafe2562105a1b99d613923c057a8a3be35aac11bdc182",
+ "frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx": "7a9e0aa7b43febf84287dbd3b15d61aa859bc7f85655674f5301fa6f2caf3b44",
+ "frontend-modern/src/components/Settings/connectionsTableModel.ts": "dacc4a4fa8c29483f03d036111d4d0583f3d5b4033362e184f6c359ccecfac6e",
+ "frontend-modern/src/components/Settings/useConnectionsLedger.ts": "19ca6b9c1c98c90a83856f6526546e647d1eb4f390dbb42f34f02a38ee0f745c",
+ "frontend-modern/src/utils/connectionErrorPresentation.ts": "c75d8dabf06db81f194c223b6b7e6bb44a3baf625ae20d930aeab14942b623b1"
+ },
+ "routes": [
+ "/settings/infrastructure",
+ "/alerts/overview"
+ ],
+ "viewports": [
+ {
+ "width": 1280,
+ "height": 800
+ },
+ {
+ "width": 375,
+ "height": 812
+ }
+ ],
+ "states": [
+ "Before (pre-change binary, same fake PMG whose qshape read returns HTTP 500): mail-gw row shows a red Unreachable badge and a red 'Server error' banner; a critical connection-degraded alert fires",
+ "After: the same row shows an amber Degraded badge, coverage Mail stats, Quarantine, Queues, and an amber banner 'Queue status could not be read: Server error. ...'",
+ "After: /api/connections reports state degraded, adapterHealth degraded, lastError.category partial",
+ "After: Alerts overview lists one connection-degraded alert at WARNING naming the pmg_queue_status read",
+ "After at 375px: the card layout shows the Degraded badge and the amber banner without clipping"
+ ],
+ "interactions": [
+ "Added the fake PMG through POST /api/config/nodes on each stack and waited for polls",
+ "Opened the mail-gw Manage drawer (editor only, no health readout) and closed it",
+ "Opened the Alerts overview and read the active alert",
+ "Resized to phone width and reloaded Settings > Infrastructure"
+ ]
+}
diff --git a/frontend-modern/src/api/__tests__/connections.test.ts b/frontend-modern/src/api/__tests__/connections.test.ts
index 8cbbb08e97..594cf66cea 100644
--- a/frontend-modern/src/api/__tests__/connections.test.ts
+++ b/frontend-modern/src/api/__tests__/connections.test.ts
@@ -64,6 +64,38 @@ describe('ConnectionsAPI', () => {
expect(result).toEqual({ connections, systems });
});
+ it('list() keeps a degraded partial-read row and its error category', async () => {
+ const connections: Connection[] = [
+ {
+ id: 'pmg:mail',
+ type: 'pmg',
+ name: 'mail',
+ address: 'https://pmg.lan:8006',
+ state: 'degraded',
+ stateReason: 'pmg_queue_status failed on mail: API error 500',
+ enabled: true,
+ surfaces: ['queues'],
+ scope: { queues: true },
+ lastSeen: '2026-10-26T12:00:00Z',
+ lastError: {
+ at: '2026-10-26T12:00:00Z',
+ message: 'pmg_queue_status failed on mail: API error 500',
+ category: 'partial',
+ },
+ source: 'manual',
+ capabilities: { supportsPause: true, supportsScope: true, supportsTest: true },
+ },
+ ];
+ mockedApiFetchJSON.mockResolvedValueOnce({ connections, systems: [] });
+
+ const result = await ConnectionsAPI.list();
+
+ expect(result.connections[0]).toMatchObject({
+ state: 'degraded',
+ lastError: { category: 'partial' },
+ });
+ });
+
it('list() normalizes missing fields to empty collections', async () => {
mockedApiFetchJSON.mockResolvedValueOnce({} as { connections?: Connection[] });
diff --git a/frontend-modern/src/api/connections.ts b/frontend-modern/src/api/connections.ts
index 747fe02767..eebef55d6d 100644
--- a/frontend-modern/src/api/connections.ts
+++ b/frontend-modern/src/api/connections.ts
@@ -5,7 +5,7 @@ export type ConnectionType =
'pve' | 'pbs' | 'pmg' | 'vmware' | 'truenas' | 'availability' | 'agent' | 'docker' | 'kubernetes';
export type ConnectionState =
- 'active' | 'paused' | 'unauthorized' | 'unreachable' | 'stale' | 'pending';
+ 'active' | 'paused' | 'unauthorized' | 'unreachable' | 'stale' | 'pending' | 'degraded';
export type ConnectionSource = 'manual' | 'agent' | 'script';
@@ -86,6 +86,9 @@ export interface ConnectionCapabilities {
export interface ConnectionError {
message: string;
at: string;
+ // "partial" marks a downstream read that failed after the connection
+ // itself answered; the connection state is then "degraded".
+ category?: string;
}
export interface ConnectionFleetGovernance {
diff --git a/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx b/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx
index 81936b216b..6fc8a7fd6c 100644
--- a/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx
+++ b/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx
@@ -225,6 +225,13 @@ interface ConfiguredSourceGroup {
rows: InfrastructureSystemRow[];
}
+// A partial read is a warning on a reachable source; any other error is an
+// outage and keeps the critical treatment.
+const rowErrorBannerClass = (row: InfrastructureSystemRow): string =>
+ row.lastErrorTone === 'warning'
+ ? 'rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-900/25 dark:bg-amber-950/25 dark:text-amber-200'
+ : 'rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200';
+
const formatCount = (count: number, noun: string): string =>
`${count} ${noun}${count === 1 ? '' : 's'}`;
@@ -1178,7 +1185,7 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
<Show when={row.lastErrorMessage}>
<div
role="alert"
- class="mt-1 rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200"
+ class={`mt-1 ${rowErrorBannerClass(row)}`}
>
{row.lastErrorMessage}
</div>
@@ -1775,10 +1782,7 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
</footer>
<Show when={row.lastErrorMessage}>
- <div
- role="alert"
- class="mt-2 rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200"
- >
+ <div role="alert" class={`mt-2 ${rowErrorBannerClass(row)}`}>
{row.lastErrorMessage}
</div>
</Show>
diff --git a/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx b/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx
index 384951e125..3452aaedab 100644
--- a/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx
+++ b/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx
@@ -366,6 +366,79 @@ describe('InfrastructureWorkspace', () => {
expect(screen.getByRole('button', { name: /^Manage$/i })).toBeInTheDocument();
});
+ it('shows a partial read as a degraded source with a warning banner naming the reading', async () => {
+ const provider = connectionFixture({ state: 'degraded' });
+ connectionState.rows = [
+ {
+ id: provider.id,
+ ownerType: provider.type,
+ name: provider.name,
+ subtitle: 'via platform API',
+ source: 'api',
+ host: provider.address,
+ coverageLabels: ['VMs'],
+ statusLabel: 'Degraded',
+ statusClassName: 'bg-yellow-100',
+ agentUpdateCount: 0,
+ lastActivityText: '1m ago',
+ lastErrorMessage: 'Queue status could not be read: Server error.',
+ lastErrorTone: 'warning',
+ ...emptyFleetRow,
+ enabled: true,
+ canEdit: true,
+ canPause: false,
+ canRemove: true,
+ isAgent: false,
+ isCluster: false,
+ attachedConnections: [],
+ members: [],
+ connection: provider,
+ },
+ ];
+ renderWorkspace();
+
+ const banner = await screen.findByText('Queue status could not be read: Server error.');
+ expect(banner.getAttribute('class')).toContain('bg-amber-50');
+ expect(banner.getAttribute('class')).not.toContain('bg-rose-50');
+ expect(screen.getAllByText('Degraded').length).toBeGreaterThan(0);
+ expect(screen.queryByText('Unreachable')).toBeNull();
+ });
+
+ it('keeps an outage error banner critical', async () => {
+ const provider = connectionFixture({ state: 'unreachable' });
+ connectionState.rows = [
+ {
+ id: provider.id,
+ ownerType: provider.type,
+ name: provider.name,
+ subtitle: 'via platform API',
+ source: 'api',
+ host: provider.address,
+ coverageLabels: ['VMs'],
+ statusLabel: 'Unreachable',
+ statusClassName: 'bg-rose-100',
+ agentUpdateCount: 0,
+ lastActivityText: '1m ago',
+ lastErrorMessage: 'Connection refused.',
+ lastErrorTone: 'critical',
+ ...emptyFleetRow,
+ enabled: true,
+ canEdit: true,
+ canPause: false,
+ canRemove: true,
+ isAgent: false,
+ isCluster: false,
+ attachedConnections: [],
+ members: [],
+ connection: provider,
+ },
+ ];
+ renderWorkspace();
+
+ const banner = await screen.findByText('Connection refused.');
+ expect(banner.getAttribute('class')).toContain('bg-rose-50');
+ });
+
it('renders the source manager landing without empty platform sections', async () => {
renderWorkspace();
diff --git a/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts b/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts
index c66e42d3ed..f4f257fdcc 100644
--- a/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts
+++ b/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts
@@ -46,6 +46,52 @@ describe('useConnectionsLedger', () => {
expect(result.rows().find((row) => row.ownerType === 'pve')?.statusLabel).toBe('Pending');
});
+ it('shows a partial read as a degraded, still-reporting connection', async () => {
+ const connections: Connection[] = [
+ {
+ id: 'pmg:mail',
+ type: 'pmg',
+ name: 'mail',
+ address: 'https://pmg.lan:8006',
+ state: 'degraded',
+ stateReason: 'pmg_queue_status failed on mail: active queue: API error 500: qshape failed',
+ enabled: true,
+ surfaces: ['mailStats', 'queues'],
+ scope: { mailStats: true, queues: true },
+ lastSeen: '2026-10-26T12:00:00Z',
+ lastError: {
+ at: '2026-10-26T12:00:00Z',
+ message: 'pmg_queue_status failed on mail: active queue: API error 500: qshape failed',
+ category: 'partial',
+ },
+ source: 'manual',
+ capabilities: { supportsPause: true, supportsScope: true, supportsTest: true },
+ fleet: {
+ enrollmentState: 'configured',
+ livenessState: 'degraded',
+ versionDrift: 'not-applicable',
+ adapterHealth: 'degraded',
+ configRollout: 'configured',
+ credentialStatus: 'verified',
+ updateStatus: 'not-applicable',
+ remoteControl: 'not-applicable',
+ },
+ },
+ ];
+ vi.spyOn(ConnectionsAPI, 'list').mockResolvedValue({ connections, systems: [] });
+
+ const { result } = renderHook(() => useConnectionsLedger());
+
+ await waitFor(() => expect(result.rows()).toHaveLength(1));
+ const row = result.rows()[0];
+ expect(row.statusLabel).toBe('Degraded');
+ expect(row.statusClassName).toContain('bg-yellow-100');
+ expect(row.source).toBe('api');
+ // Name the missing reading, in warning tone: the source still answers.
+ expect(row.lastErrorMessage).toMatch(/^Queue status could not be read: Server error\./);
+ expect(row.lastErrorTone).toBe('warning');
+ });
+
it('renders standalone agent rows with compact host identity and endpoint context', async () => {
const connections: Connection[] = [
{
diff --git a/frontend-modern/src/components/Settings/connectionsTableModel.ts b/frontend-modern/src/components/Settings/connectionsTableModel.ts
index 6b9aad8e6e..04c6a99a8d 100644
--- a/frontend-modern/src/components/Settings/connectionsTableModel.ts
+++ b/frontend-modern/src/components/Settings/connectionsTableModel.ts
@@ -443,6 +443,13 @@ const livenessSignal = (state: ConnectionFleetLivenessState): FleetGovernanceSig
detail: 'Pulse has not seen recent activity from this source.',
tone: 'warning',
};
+ case 'degraded':
+ return {
+ key: 'liveness',
+ label: 'Degraded',
+ detail: 'Pulse reaches this source, but some of its readings failed.',
+ tone: 'warning',
+ };
case 'pending':
return {
key: 'liveness',
@@ -1030,6 +1037,9 @@ export interface InfrastructureSystemRow {
agentUpdateCount: number;
lastActivityText: string;
lastErrorMessage?: string;
+ // A partial read (some readings failed on a reachable source) is a warning,
+ // not an outage.
+ lastErrorTone?: 'warning' | 'critical';
fleetSignals: FleetGovernanceSignal[];
fleetHighlights: FleetGovernanceSignal[];
problem?: InfrastructureRowProblem;
diff --git a/frontend-modern/src/components/Settings/useConnectionsLedger.ts b/frontend-modern/src/components/Settings/useConnectionsLedger.ts
index a8b382aa61..45623ea45d 100644
--- a/frontend-modern/src/components/Settings/useConnectionsLedger.ts
+++ b/frontend-modern/src/components/Settings/useConnectionsLedger.ts
@@ -1,6 +1,9 @@
import { createMemo } from 'solid-js';
import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';
-import { formatConnectionErrorMessage } from '@/utils/connectionErrorPresentation';
+import {
+ formatConnectionErrorMessage,
+ formatPartialReadErrorMessage,
+} from '@/utils/connectionErrorPresentation';
import { AGENT_WAITING_FOR_FIRST_REPORT_LABEL } from '@/utils/agentResources';
import {
ConnectionsAPI,
@@ -64,6 +67,10 @@ const STATE_PRESENTATION: Record<ConnectionState, { label: string; badgeClass: s
label: 'Stale',
badgeClass: 'bg-yellow-100 text-yellow-800 dark:bg-yellow-900/25 dark:text-yellow-200',
},
+ degraded: {
+ label: 'Degraded',
+ badgeClass: 'bg-yellow-100 text-yellow-800 dark:bg-yellow-900/25 dark:text-yellow-200',
+ },
pending: {
label: 'Pending',
badgeClass: 'bg-surface-alt text-base-content',
@@ -91,9 +98,10 @@ const CONNECTION_STATE_SEVERITY: Record<ConnectionState, number> = {
active: 0,
paused: 1,
pending: 2,
- stale: 3,
- unauthorized: 4,
- unreachable: 5,
+ degraded: 3,
+ stale: 4,
+ unauthorized: 5,
+ unreachable: 6,
};
const coverageLabelsFor = (connections: readonly Connection[]): string[] => {
@@ -140,9 +148,10 @@ const subtitleFor = (connections: readonly Connection[], primaryConnection: Conn
// A connection is "contributing" if it is currently delivering data — not
// merely configured. Otherwise the source badge claims "API + Agent" when
// the agent has been silent for hours, which directly contradicts the
-// "Agent offline" chip on the same row.
+// "Agent offline" chip on the same row. A degraded connection still
+// delivers every reading except the one that failed.
const isContributing = (connection: Connection): boolean =>
- connection.state === 'active' || connection.state === 'paused';
+ connection.state === 'active' || connection.state === 'paused' || connection.state === 'degraded';
const sourceFor = (connections: readonly Connection[]): InfrastructureSourceKind => {
const contributors = connections.filter(isContributing);
@@ -364,10 +373,19 @@ const buildRow = (
// Run raw backend error chains through the user-facing presenter so the
// connection table doesn't surface implementation strings like 'context
// deadline exceeded' or 'Client.Timeout exceeded while awaiting headers'.
- const rawLastErrorMessage =
- primaryConnection.lastError?.message ??
- attachedConnections.find((connection) => connection.lastError?.message)?.lastError?.message;
- const lastErrorMessage = formatConnectionErrorMessage(rawLastErrorMessage) ?? undefined;
+ const lastError = primaryConnection.lastError?.message
+ ? primaryConnection.lastError
+ : attachedConnections.find((connection) => connection.lastError?.message)?.lastError;
+ const partialRead = lastError?.category === 'partial';
+ const lastErrorMessage =
+ (partialRead
+ ? formatPartialReadErrorMessage(lastError?.message)
+ : formatConnectionErrorMessage(lastError?.message)) ?? undefined;
+ const lastErrorTone: InfrastructureSystemRow['lastErrorTone'] = lastErrorMessage
+ ? partialRead
+ ? 'warning'
+ : 'critical'
+ : undefined;
const hostTelemetryLabel = surfaceLabel('host');
const coverageLabels = isCluster
? coverageLabelsFor(componentConnections).filter((label) => label !== hostTelemetryLabel)
@@ -426,6 +444,7 @@ const buildRow = (
? lastActivityTextFromLastSeen(rollup.lastSeen)
: connectionLastActivityText(primaryConnection),
lastErrorMessage,
+ lastErrorTone,
fleetSignals,
fleetHighlights,
problem:
diff --git a/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts b/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts
index 46a2b78212..31fe332be7 100644
--- a/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts
+++ b/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
import {
formatConnectionErrorMessage,
+ formatPartialReadErrorMessage,
humanizeConnectionError,
} from '../connectionErrorPresentation';
@@ -170,3 +171,35 @@ describe('humanizeConnectionError', () => {
expect(formatted).toBe('a totally unknown failure mode');
});
});
+
+describe('formatPartialReadErrorMessage', () => {
+ it('names the failed reading before the cause', () => {
+ expect(
+ formatPartialReadErrorMessage(
+ 'pmg_queue_status failed on mail-gw: active queue: API error 500: {"errors":"qshape failed"}',
+ ),
+ ).toBe(
+ 'Queue status could not be read: Server error. The platform returned a server error. Check the platform logs for the underlying cause.',
+ );
+ });
+
+ it('counts the other failed readings', () => {
+ expect(
+ formatPartialReadErrorMessage(
+ [
+ 'pmg_spam_quarantine failed on mail-gw: API error 403: Permission check failed',
+ 'pmg_virus_quarantine failed on mail-gw: API error 403: Permission check failed',
+ 'pmg_queue_status failed on mail-gw: API error 500',
+ ].join('\n'),
+ ),
+ ).toMatch(/^Spam quarantine and 2 other readings could not be read: Permission denied\./);
+ });
+
+ it('falls back to a generic subject without an operation prefix', () => {
+ expect(formatPartialReadErrorMessage('context deadline exceeded')).toMatch(
+ /^Some readings could not be read: Connection timed out\./,
+ );
+ expect(formatPartialReadErrorMessage('')).toBeNull();
+ expect(formatPartialReadErrorMessage(undefined)).toBeNull();
+ });
+});
diff --git a/frontend-modern/src/utils/connectionErrorPresentation.ts b/frontend-modern/src/utils/connectionErrorPresentation.ts
index 5746cb78b6..de202f1caa 100644
--- a/frontend-modern/src/utils/connectionErrorPresentation.ts
+++ b/frontend-modern/src/utils/connectionErrorPresentation.ts
@@ -160,3 +160,38 @@ export const formatConnectionErrorMessage = (raw?: string | null): string | null
if (!humanized) return null;
return humanized.hint ? `${humanized.headline}. ${humanized.hint}` : humanized.headline;
};
+
+// `pmg_queue_status failed on mail-gw: ...`: the monitor's read operation,
+// the instance, then the cause.
+const READ_OPERATION_PATTERN = /^([a-z]+(?:_[a-z]+)+)\s+failed\s+on\s+[^:]+:\s*/i;
+
+const readOperationLabel = (operation: string): string => {
+ const words = operation
+ .toLowerCase()
+ .replace(/^(?:pmg|pbs|pve)_/, '')
+ .replace(/_/g, ' ');
+ return words.charAt(0).toUpperCase() + words.slice(1);
+};
+
+/**
+ * Render a partial-read error: the connection answered, but some readings
+ * failed (one per line). The generic cause alone ('Server error') hides which
+ * data is missing, so name the failed reading first.
+ */
+export const formatPartialReadErrorMessage = (raw?: string | null): string | null => {
+ const failures = (raw ?? '')
+ .split('\n')
+ .map((line) => line.trim())
+ .filter(Boolean);
+ const first = failures[0];
+ if (!first) return null;
+ const match = first.match(READ_OPERATION_PATTERN);
+ const label = match ? readOperationLabel(match[1]) : 'Some readings';
+ const cause = formatConnectionErrorMessage(match ? first.slice(match[0].length) : first);
+ const others = failures.length - 1;
+ const subject =
+ match && others > 0
+ ? `${label} and ${others} other ${others === 1 ? 'reading' : 'readings'}`
+ : label;
+ return cause ? `${subject} could not be read: ${cause}` : `${subject} could not be read.`;
+};
diff --git a/internal/alerts/connection.go b/internal/alerts/connection.go
index 4d3fc7ccc7..17e6a8963d 100644
--- a/internal/alerts/connection.go
+++ b/internal/alerts/connection.go
@@ -24,6 +24,7 @@ const (
ConnectionStateUnauthorized ConnectionState = "unauthorized"
ConnectionStatePaused ConnectionState = "paused"
ConnectionStatePending ConnectionState = "pending"
+ ConnectionStateDegraded ConnectionState = "degraded"
)
// ConnectionType narrows the set of connections that participate in the
@@ -181,8 +182,9 @@ func (m *Manager) clearConnectionDegradedIntentNoLock(resourceID string) {
}
// CheckConnection raises or clears the connection-degraded alert for one
-// platform connection. Severity scales with observed state: stale → warning,
-// unreachable / unauthorized → critical. State=active runs through the
+// platform connection. Severity scales with observed state: stale and degraded
+// (reachable, but some readings failed) → warning, unreachable / unauthorized
+// → critical. State=active runs through the
// recovery-confirmation gate before resolving. Paused, disabled, and
// non-platform connections are skipped.
func (m *Manager) CheckConnection(snap ConnectionSnapshot) {
@@ -204,7 +206,7 @@ func (m *Manager) CheckConnection(snap ConnectionSnapshot) {
case ConnectionStateActive:
m.clearConnectionDegradedAlert(snap)
return
- case ConnectionStateStale, ConnectionStateUnreachable, ConnectionStateUnauthorized:
+ case ConnectionStateStale, ConnectionStateDegraded, ConnectionStateUnreachable, ConnectionStateUnauthorized:
// fall through and fire
default:
// Pending/unknown breaks both confirmation runs without resolving an
@@ -233,6 +235,7 @@ func (m *Manager) CheckConnection(snap ConnectionSnapshot) {
connectionDegradedStateKey,
[]string{
string(ConnectionStateStale),
+ string(ConnectionStateDegraded),
string(ConnectionStateUnreachable),
string(ConnectionStateUnauthorized),
},
diff --git a/internal/alerts/operational_contract_test.go b/internal/alerts/operational_contract_test.go
index abdef56a18..2d6bccd7bf 100644
--- a/internal/alerts/operational_contract_test.go
+++ b/internal/alerts/operational_contract_test.go
@@ -949,6 +949,66 @@ func TestConnectionDegradedAlertNamesItsPlatform(t *testing.T) {
}
}
+// A partial platform read (the connection answered, one reading failed) is a
+// warning-level degraded observation, never an outage (#2960).
+func TestConnectionDegradedPartialReadAlertsAtWarning(t *testing.T) {
+ t.Run("degraded fires a warning, never critical", func(t *testing.T) {
+ m := newTestManager(t)
+ snap := platformConnectionSnapshot("conn-partial", "Mail gateway", ConnectionStateDegraded)
+ snap.Type = ConnectionTypePMG
+ snap.StateReason = "pmg_queue_status: api error 500"
+ snap.LastError = &ConnectionErrorSnapshot{At: time.Now(), Message: snap.StateReason, Category: "partial"}
+ alertID := canonicalDiscreteStateStateID(snap.ID, connectionDegradedStateKey)
+
+ m.CheckConnection(snap)
+ m.CheckConnection(snap)
+ if testHasActiveAlert(t, m, alertID) {
+ t.Fatal("expected no alert before three degraded observations")
+ }
+ m.CheckConnection(snap)
+
+ alert := testRequireActiveAlert(t, m, alertID)
+ if alert.Level != AlertLevelWarning {
+ t.Errorf("expected warning level for degraded, got %s", alert.Level)
+ }
+ if got, _ := alert.Metadata["state"].(string); got != string(ConnectionStateDegraded) {
+ t.Errorf("expected state=degraded in metadata, got %q", got)
+ }
+ if got, _ := alert.Metadata["lastErrorCategory"].(string); got != "partial" {
+ t.Errorf("expected lastErrorCategory=partial in metadata, got %q", got)
+ }
+ })
+
+ t.Run("degraded keeps an unreachable outage open at warning", func(t *testing.T) {
+ m := newTestManager(t)
+ snap := platformConnectionSnapshot("conn-partial-recover", "Mail gateway", ConnectionStateUnreachable)
+ snap.Type = ConnectionTypePMG
+ alertID := canonicalDiscreteStateStateID(snap.ID, connectionDegradedStateKey)
+
+ m.CheckConnection(snap)
+ m.CheckConnection(snap)
+ m.CheckConnection(snap)
+ if alert := testRequireActiveAlert(t, m, alertID); alert.Level != AlertLevelCritical {
+ t.Fatalf("expected critical outage before the partial poll, got %s", alert.Level)
+ }
+
+ // The connection answers again but one reading still fails: a
+ // missing reading is not recovery evidence.
+ snap.State = ConnectionStateDegraded
+ snap.StateReason = "pmg_queue_status: api error 500"
+ for i := 0; i < 5; i++ {
+ m.CheckConnection(snap)
+ }
+ alert := testRequireActiveAlert(t, m, alertID)
+ if alert.Level != AlertLevelWarning {
+ t.Errorf("expected the open alert to drop to warning, got %s", alert.Level)
+ }
+ if got, _ := alert.Metadata["state"].(string); got != string(ConnectionStateDegraded) {
+ t.Errorf("expected state=degraded in metadata, got %q", got)
+ }
+ })
+}
+
func TestCapacityForecastAlertNamesItsPlatform(t *testing.T) {
m := newTestManager(t)
m.ClearActiveAlerts()
diff --git a/internal/api/connections_aggregator.go b/internal/api/connections_aggregator.go
index 8dd63aa70e..c821de8f70 100644
--- a/internal/api/connections_aggregator.go
+++ b/internal/api/connections_aggregator.go
@@ -769,7 +769,7 @@ func connectionFleetAdapterHealth(conn Connection) string {
switch conn.State {
case ConnectionStateActive:
return fleetStateHealthy
- case ConnectionStateStale, ConnectionStatePending:
+ case ConnectionStateStale, ConnectionStatePending, ConnectionStateDegraded:
return fleetStateDegraded
case ConnectionStateUnauthorized, ConnectionStateUnreachable:
return fleetStateBlocked
@@ -1459,6 +1459,18 @@ func deriveConnectionState(enabled bool, h monitoring.InstanceHealth, now time.T
return ConnectionStatePending, "awaiting first poll", nil, nil
}
+ // A partial error means the poll reached the connection, authenticated and
+ // read its version; only a downstream read failed (#2960). The connection
+ // is reachable and its other readings are fresh, so it is degraded, not
+ // unreachable, and a 403 from one endpoint does not make the credential
+ // unauthorized. A connection that also stopped polling is still stale.
+ if lastError != nil && lastError.Category == monitoring.PollErrorCategoryPartial {
+ if lastSeen != nil && now.Sub(*lastSeen) > connectionStaleThresholdFor(pollInterval) {
+ return ConnectionStateStale, fmt.Sprintf("no successful poll in %s", now.Sub(*lastSeen).Round(time.Second)), lastSeen, lastError
+ }
+ return ConnectionStateDegraded, lastError.Message, lastSeen, lastError
+ }
+
if lastError != nil && connectionAuthErrorPattern.MatchString(lastError.Message) {
return ConnectionStateUnauthorized, lastError.Message, lastSeen, lastError
}
diff --git a/internal/api/connections_aggregator_test.go b/internal/api/connections_aggregator_test.go
index 4af4841114..ecb74087c1 100644
--- a/internal/api/connections_aggregator_test.go
+++ b/internal/api/connections_aggregator_test.go
@@ -107,6 +107,78 @@ func TestDeriveConnectionState_Stale(t *testing.T) {
}
}
+// A partial poll error (#2960) means the connection answered and only one
+// downstream read failed: degraded, never unreachable or unauthorized.
+func TestDeriveConnectionState_PartialReadIsDegraded(t *testing.T) {
+ now := time.Now()
+ for _, message := range []string{
+ "pmg_queue_status: api error 500: queue status unavailable",
+ "pmg_quarantine: api error 403: Permission check failed",
+ } {
+ h := healthEntry(ptrTime(now.Add(-10*time.Second)), message, monitoring.PollErrorCategoryPartial, "closed")
+ state, reason, lastSeen, lastError := deriveConnectionState(true, h, now, 0)
+ if state != ConnectionStateDegraded {
+ t.Fatalf("%q: got %q, want %q", message, state, ConnectionStateDegraded)
+ }
+ if reason != message {
+ t.Fatalf("%q: reason = %q, want the failed read", message, reason)
+ }
+ if lastSeen == nil {
+ t.Fatalf("%q: expected lastSeen from the partial poll's success", message)
+ }
+ if lastError == nil || lastError.Category != monitoring.PollErrorCategoryPartial || lastError.Message != message {
+ t.Fatalf("%q: lastError = %+v, want the partial read kept visible", message, lastError)
+ }
+ }
+}
+
+func TestDeriveConnectionState_PartialReadStillGoesStale(t *testing.T) {
+ now := time.Now()
+ h := healthEntry(ptrTime(now.Add(-5*time.Minute)), "pmg_queue_status: api error 500", monitoring.PollErrorCategoryPartial, "closed")
+ state, _, _, lastError := deriveConnectionState(true, h, now, 0)
+ if state != ConnectionStateStale {
+ t.Fatalf("got %q, want %q", state, ConnectionStateStale)
+ }
+ if lastError == nil || lastError.Category != monitoring.PollErrorCategoryPartial {
+ t.Fatalf("lastError = %+v, want the partial read kept visible", lastError)
+ }
+}
+
+func TestBuildConnections_PMGPartialReadIsDegradedNotBlocked(t *testing.T) {
+ now := time.Now()
+ got := buildConnections(aggregatorInputs{
+ pmgInstances: []config.PMGInstance{{Name: "mail-main", Host: "https://pmg.lan:8006", User: "root@pam", Password: "secret"}},
+ instanceHealth: map[string]monitoring.InstanceHealth{
+ "pmg::mail-main": healthEntry(ptrTime(now.Add(-10*time.Second)), "pmg_quarantine: api error 403: Permission check failed", monitoring.PollErrorCategoryPartial, "closed"),
+ },
+ now: now,
+ })
+ if len(got) != 1 {
+ t.Fatalf("expected 1 connection, got %d", len(got))
+ }
+ conn := got[0]
+ if conn.State != ConnectionStateDegraded {
+ t.Fatalf("state = %q, want %q", conn.State, ConnectionStateDegraded)
+ }
+ if conn.Fleet.AdapterHealth != fleetStateDegraded {
+ t.Fatalf("adapter health = %q, want %q", conn.Fleet.AdapterHealth, fleetStateDegraded)
+ }
+ if conn.Fleet.CredentialStatus != fleetStateVerified {
+ t.Fatalf("credential status = %q, want %q: one endpoint's 403 is not a rejected credential", conn.Fleet.CredentialStatus, fleetStateVerified)
+ }
+ if conn.Fleet.Rollout != nil && conn.Fleet.Rollout.Status == fleetRolloutBlocked {
+ t.Fatalf("rollout = %+v, a reachable connection must not be blocked", conn.Fleet.Rollout)
+ }
+
+ snaps := snapshotConnectionsForAlerts(got)
+ if len(snaps) != 1 || snaps[0].State != alerts.ConnectionStateDegraded {
+ t.Fatalf("alert snapshots = %+v, want one degraded snapshot", snaps)
+ }
+ if snaps[0].LastError == nil || snaps[0].LastError.Category != monitoring.PollErrorCategoryPartial {
+ t.Fatalf("alert snapshot lastError = %+v, want the partial read", snaps[0].LastError)
+ }
+}
+
func TestDeriveConnectionState_Active(t *testing.T) {
now := time.Now()
h := healthEntry(ptrTime(now.Add(-10*time.Second)), "", "", "closed")
diff --git a/internal/api/connections_grouping.go b/internal/api/connections_grouping.go
index 1597325242..b8e8d80b84 100644
--- a/internal/api/connections_grouping.go
+++ b/internal/api/connections_grouping.go
@@ -415,12 +415,14 @@ func connectionSystemMemberStateSeverity(state ConnectionState) int {
return 1
case ConnectionStatePending:
return 2
- case ConnectionStateStale:
+ case ConnectionStateDegraded:
return 3
- case ConnectionStateUnauthorized:
+ case ConnectionStateStale:
return 4
- case ConnectionStateUnreachable:
+ case ConnectionStateUnauthorized:
return 5
+ case ConnectionStateUnreachable:
+ return 6
default:
return 2
}
diff --git a/internal/api/connections_types.go b/internal/api/connections_types.go
index a9d560514d..f7b31ca5aa 100644
--- a/internal/api/connections_types.go
+++ b/internal/api/connections_types.go
@@ -14,6 +14,9 @@ const (
ConnectionStateUnreachable ConnectionState = "unreachable"
ConnectionStateStale ConnectionState = "stale"
ConnectionStatePending ConnectionState = "pending"
+ // ConnectionStateDegraded is a reachable connection whose latest poll
+ // left some readings unread (a poll error with category "partial").
+ ConnectionStateDegraded ConnectionState = "degraded"
)
// ConnectionType is the product family that owns the connection. The value is
diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go
index 3f88f18416..96f9c3982c 100644
--- a/internal/api/contract_test.go
+++ b/internal/api/contract_test.go
@@ -18327,6 +18327,52 @@ func TestContract_ConnectionPayloadShapeStaysCanonical(t *testing.T) {
assertJSONSnapshot(t, body, want)
}
+// TestContract_ConnectionPartialReadPayloadIsDegraded pins the wire shape of a
+// platform connection whose poll answered but left one reading unread (#2960):
+// state "degraded", the failed read as stateReason, and lastError.category
+// "partial", with credentials still verified.
+func TestContract_ConnectionPartialReadPayloadIsDegraded(t *testing.T) {
+ now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
+ lastSuccess := now.Add(-10 * time.Second)
+ message := "pmg_queue_status failed on mail: API error 500"
+ conn := buildPMGConnection(config.PMGInstance{Name: "mail", Host: "https://pmg.lan:8006", User: "root@pam", Password: "secret"},
+ map[string]monitoring.InstanceHealth{
+ "pmg::mail": {
+ PollStatus: monitoring.InstancePollStatus{
+ LastSuccess: &lastSuccess,
+ LastError: &monitoring.ErrorDetail{At: lastSuccess, Message: message, Category: monitoring.PollErrorCategoryPartial},
+ },
+ Breaker: monitoring.InstanceBreaker{State: "closed"},
+ },
+ }, now, 0)
+ body, err := json.Marshal(conn)
+ if err != nil {
+ t.Fatalf("marshal Connection: %v", err)
+ }
+ var got struct {
+ State string `json:"state"`
+ StateReason string `json:"stateReason"`
+ LastError struct {
+ Message string `json:"message"`
+ Category string `json:"category"`
+ } `json:"lastError"`
+ Fleet struct {
+ LivenessState string `json:"livenessState"`
+ AdapterHealth string `json:"adapterHealth"`
+ CredentialStatus string `json:"credentialStatus"`
+ } `json:"fleet"`
+ }
+ if err := json.Unmarshal(body, &got); err != nil {
+ t.Fatalf("unmarshal Connection: %v", err)
+ }
+ if got.State != "degraded" || got.StateReason != message || got.LastError.Message != message || got.LastError.Category != "partial" {
+ t.Fatalf("partial read payload = %s", body)
+ }
+ if got.Fleet.LivenessState != "degraded" || got.Fleet.AdapterHealth != "degraded" || got.Fleet.CredentialStatus != "verified" {
+ t.Fatalf("partial read fleet payload = %s", body)
+ }
+}
+
func TestContract_ConnectionSystemMembersPayloadShapeStaysCanonical(t *testing.T) {
system := ConnectionSystem{
ID: "pve-lab",
diff --git a/internal/monitoring/monitor_additional_test.go b/internal/monitoring/monitor_additional_test.go
index dd78c7cbb1..14dbaadbca 100644
--- a/internal/monitoring/monitor_additional_test.go
+++ b/internal/monitoring/monitor_additional_test.go
@@ -1910,7 +1910,7 @@ func TestRecordTaskOutcomeReadErrorSchedulesAsSuccess(t *testing.T) {
}
status := m.pollStatusMap[key]
if status.ConsecutiveFailures != 0 || status.LastSuccess.IsZero() || !status.FirstFailureAt.IsZero() ||
- status.LastErrorCategory != pollReadErrorCategory || !strings.Contains(status.LastErrorMessage, "pmg_queue_status") ||
+ status.LastErrorCategory != PollErrorCategoryPartial || !strings.Contains(status.LastErrorMessage, "pmg_queue_status") ||
!status.LastErrorAt.Equal(status.LastSuccess) {
t.Fatalf("read error not kept as the outstanding poll error: %+v", status)
}
diff --git a/internal/monitoring/poll_error_category.go b/internal/monitoring/poll_error_category.go
new file mode 100644
index 0000000000..b762b8d783
--- /dev/null
+++ b/internal/monitoring/poll_error_category.go
@@ -0,0 +1,7 @@
+package monitoring
+
+// PollErrorCategoryPartial is the ErrorDetail.Category recordTaskOutcome sets
+// when a poll reached the connection but left some readings unread. The
+// connections ledger reads it to present a degraded, still-reachable
+// connection instead of an outage (#2960).
+const PollErrorCategoryPartial = pollReadErrorCategory