Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain guest API readings without inventing freshness
pulse-triage[bot]3e4596b92ac4
diff --git a/docs/API.md b/docs/API.mdindex 296d4cc116..b437b434d5 100644--- a/docs/API.md+++ b/docs/API.md@@ -272,7 +272,45 @@ canonical `verdict` and stable `reasons` (`code` plus optional compact`detail`). Consumers should use this envelope for cross-platform postureinstead of deriving health from provider-specific status strings.-Note: guest disk usage percentages use `-1` as an "unknown" sentinel — reported when a VM is stopped or its guest agent is unavailable, so there is no filesystem view to measure. Consumers should treat negative values as "no data", not as a percentage; the accompanying `diskStatusReason` field (e.g. `vm-stopped`, `agent-disabled`) says why.+#### Guest readings: availability and age++In legacy VM payloads, `disk.usage: -1` means unavailable, not negative usage.+In `GET /api/resources` and `GET /api/resources/{id}`, an unavailable disk+reading is omitted as `metrics.disk`. **Do not replace an absent metric with+zero**; a genuinely observed zero is valid. Allocated virtual-disk capacity+alone does not establish used space inside the guest.++Read `proxmox.diskStatusReason` alongside the value. A `prev-` reason means+retained disk evidence, not a fresh reading. `agent-not-running` can also+represent a general guest-agent HTTP 500 error: it does not establish that the+guest's service is absent or stopped. An absent reason alone does not prove+freshness either.++For memory, use `metrics.memory.observation` for the **selected metric's**+source and age, not a different source's retained `proxmox.memory` facet:++| Observation `state` | Meaning for the consumer |+| --- | --- |+| `current` | Accepted source observation; a normal cache hit can keep the original `observedAt`. Check its age for your use. |+| `last-known` | Retained evidence, not a new measurement. Keep its original age visible; do not record it as a fresh sample. |+| `unavailable` | No usable usage observation; do not treat retained numbers or zero-valued fields as a measurement. |+| Missing or unrecognised | Freshness is unknown, including older payloads without an observation. Do not infer `current`. |++`observedAt` belongs to that source observation. If it is absent or in the+future, its age is unknown; do not substitute the resource's `lastSeen` or+`updatedAt`. In the raw `proxmox.memory` facet, `usageUnavailable: true` can+preserve known capacity without measured usage. An absent `metrics.memory`+is not 0% usage. Conversely, an independently observed PVE or Pulse-agent+memory reading can remain current while QEMU disk collection is deferred;+disk state is not memory provenance.++A running VM, an advancing row timestamp or an OK backup does not prove fresh+guest readings or successful thaw. Do not install or restart an agent or send+live guest-agent probes merely to fill these gaps, especially during a backup,+freeze/thaw or an unresponsive-guest incident. Follow [Backup safety](VM_DISK_MONITORING.md#backup-safety)+and [missing-reading guidance](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis).+Recovery needs independent thaw, fresh successful writes to every covered+filesystem and workload liveness; retained API values cannot establish it.Availability is an additive resource facet. `availability` is the compatibilitysummary used by existing clients; `availabilityChecks` contains every checkdiff --git a/frontend-modern/public/docs/API.md b/frontend-modern/public/docs/API.mdindex 296d4cc116..b437b434d5 100644--- a/frontend-modern/public/docs/API.md+++ b/frontend-modern/public/docs/API.md@@ -272,7 +272,45 @@ canonical `verdict` and stable `reasons` (`code` plus optional compact`detail`). Consumers should use this envelope for cross-platform postureinstead of deriving health from provider-specific status strings.-Note: guest disk usage percentages use `-1` as an "unknown" sentinel — reported when a VM is stopped or its guest agent is unavailable, so there is no filesystem view to measure. Consumers should treat negative values as "no data", not as a percentage; the accompanying `diskStatusReason` field (e.g. `vm-stopped`, `agent-disabled`) says why.+#### Guest readings: availability and age++In legacy VM payloads, `disk.usage: -1` means unavailable, not negative usage.+In `GET /api/resources` and `GET /api/resources/{id}`, an unavailable disk+reading is omitted as `metrics.disk`. **Do not replace an absent metric with+zero**; a genuinely observed zero is valid. Allocated virtual-disk capacity+alone does not establish used space inside the guest.++Read `proxmox.diskStatusReason` alongside the value. A `prev-` reason means+retained disk evidence, not a fresh reading. `agent-not-running` can also+represent a general guest-agent HTTP 500 error: it does not establish that the+guest's service is absent or stopped. An absent reason alone does not prove+freshness either.++For memory, use `metrics.memory.observation` for the **selected metric's**+source and age, not a different source's retained `proxmox.memory` facet:++| Observation `state` | Meaning for the consumer |+| --- | --- |+| `current` | Accepted source observation; a normal cache hit can keep the original `observedAt`. Check its age for your use. |+| `last-known` | Retained evidence, not a new measurement. Keep its original age visible; do not record it as a fresh sample. |+| `unavailable` | No usable usage observation; do not treat retained numbers or zero-valued fields as a measurement. |+| Missing or unrecognised | Freshness is unknown, including older payloads without an observation. Do not infer `current`. |++`observedAt` belongs to that source observation. If it is absent or in the+future, its age is unknown; do not substitute the resource's `lastSeen` or+`updatedAt`. In the raw `proxmox.memory` facet, `usageUnavailable: true` can+preserve known capacity without measured usage. An absent `metrics.memory`+is not 0% usage. Conversely, an independently observed PVE or Pulse-agent+memory reading can remain current while QEMU disk collection is deferred;+disk state is not memory provenance.++A running VM, an advancing row timestamp or an OK backup does not prove fresh+guest readings or successful thaw. Do not install or restart an agent or send+live guest-agent probes merely to fill these gaps, especially during a backup,+freeze/thaw or an unresponsive-guest incident. Follow [Backup safety](VM_DISK_MONITORING.md#backup-safety)+and [missing-reading guidance](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis).+Recovery needs independent thaw, fresh successful writes to every covered+filesystem and workload liveness; retained API values cannot establish it.Availability is an additive resource facet. `availability` is the compatibilitysummary used by existing clients; `availabilityChecks` contains every checkdiff --git a/frontend-modern/src/utils/__tests__/guestObservationHelp.test.ts b/frontend-modern/src/utils/__tests__/guestObservationHelp.test.tsnew file mode 100644index 0000000000..893f096b3f--- /dev/null+++ b/frontend-modern/src/utils/__tests__/guestObservationHelp.test.ts@@ -0,0 +1,101 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '@/features/docs/docMarkdown';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const read = (name: string) => readFileSync(path.join(root, name), 'utf8');+const api = read('docs/API.md');+const section =+ api.split('#### Guest readings: availability and age')[1]?.split('\nAvailability is')[0] ?? '';++function render(markdown: string, name: string): HTMLElement {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(markdown, name);+ return article;+}++const rendered = () => render(section, 'API');+const text = () => rendered().textContent?.replace(/\s+/g, ' ');++describe('guest observation API help', () => {+ it('distinguishes the legacy sentinel from an omitted unified metric and an observed zero', () => {+ expect(read('frontend-modern/public/docs/API.md')).toBe(api);+ expect(text()).toContain('disk.usage: -1 means unavailable, not negative usage');+ expect(text()).toContain('omitted as metrics.disk');+ expect(text()).toContain('Do not replace an absent metric with zero');+ expect(text()).toContain('a genuinely observed zero is valid');+ expect(text()).toContain('Allocated virtual-disk capacity alone does not establish used space');+ // These are the production omission/presence rules, not invented client fields.+ const types = read('internal/unifiedresources/types.go');+ expect(types).toMatch(/Metrics\s+\*ResourceMetrics\s+`json:"metrics,omitempty"`/);+ expect(types).toMatch(/Disk\s+\*MetricValue\s+`json:"disk,omitempty"`/);+ const metrics = read('internal/unifiedresources/metrics.go');+ expect(metrics).toContain('if hasObservedDiskUsage(disk)');+ expect(metrics).toContain('disk.Usage >= 0 && disk.Usage <= 100');+ });++ it('keeps retained disk evidence and an ambiguous error separate from a service diagnosis', () => {+ expect(text()).toContain('proxmox.diskStatusReason');+ expect(text()).toContain('A prev- reason means retained disk evidence, not a fresh reading');+ expect(text()).toContain('general guest-agent HTTP 500 error');+ expect(text()).toContain("does not establish that the guest's service is absent or stopped");+ expect(text()).toContain('An absent reason alone does not prove freshness');+ const collector = read('internal/monitoring/guest_disk_stability.go');+ expect(collector).toContain('"prev-" + diskStatusReason');+ expect(collector).toMatch(+ /case strings.Contains\(errStr, "500"\):\s+return "agent-not-running"/,+ );+ });++ it('renders selected-source memory states without renewing old or missing observations', () => {+ const rows = [...rendered().querySelectorAll('tbody tr')].map((row) =>+ [...row.querySelectorAll('td')].map((cell) => cell.textContent?.trim()),+ );+ expect(rows.map((row) => row[0])).toEqual([+ 'current',+ 'last-known',+ 'unavailable',+ 'Missing or unrecognised',+ ]);+ expect(rows[0][1]).toContain('original observedAt');+ expect(rows[1][1]).toContain('do not record it as a fresh sample');+ expect(rows[2][1]).toContain(+ 'do not treat retained numbers or zero-valued fields as a measurement',+ );+ expect(rows[3][1]).toContain('Do not infer current');+ expect(text()).toContain('metrics.memory.observation');+ expect(text()).toContain("selected metric's source and age");+ expect(text()).toContain("do not substitute the resource's lastSeen or updatedAt");+ expect(text()).toContain('usageUnavailable: true');+ expect(text()).toContain('An absent metrics.memory is not 0% usage');+ expect(text()).toContain('disk state is not memory provenance');+ const model = read('internal/models/models.go');+ expect(model).toContain('`json:"observation,omitzero"`');+ expect(model).toContain('`json:"observedAt,omitzero"`');+ expect(model).toContain('`json:"usageUnavailable,omitempty"`');+ expect(read('internal/unifiedresources/metrics.go')).toContain(+ 'Observation: memory.Observation',+ );+ });++ it('makes the existing safety and missing-reading guidance reachable without live probe commands', () => {+ expect(text()).toContain('an OK backup does not prove fresh guest readings or successful thaw');+ expect(text()).toContain('Do not install or restart an agent or send live guest-agent probes');+ expect(text()).toContain(+ 'fresh successful writes to every covered filesystem and workload liveness',+ );+ expect(text()).toContain('retained API values cannot establish it');+ expect(rendered().querySelector('pre')).toBeNull();+ const target = render(read('docs/VM_DISK_MONITORING.md'), 'VM_DISK_MONITORING');+ for (const [label, fragment] of [+ ['Backup safety', 'backup-safety'],+ ['missing-reading guidance', 'a-missing-reading-is-not-an-installation-diagnosis'],+ ]) {+ const link = rendered().querySelector(`a[href="/docs/VM_DISK_MONITORING#${fragment}"]`);+ expect(link?.textContent).toBe(label);+ expect(target.querySelector(`#${fragment}`)).not.toBeNull();+ }+ });+});