Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep Podman CPU reports on their measured interval
pulse-triage[bot]0c88b83fed81
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 8311a06f92..51a31e9ad2 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -59,18 +59,19 @@ system-information probe; no synthetic success establishes installed acceptance.### Podman interval CPU evidence — issue #2447-Every one-shot container stats response retains a cumulative CPU baseline,-even when a positive Podman compatibility percentage takes precedence. Zero,-absent or invalid percentages use the valid nanosecond/wall-clock interval,-without Docker's host-core multiplier. First samples, counter resets and changes-of container start time rebaseline rather than presenting lifetime CPU as current-usage. Repeated or older read timestamps cannot move the baseline backwards.-Positive native percentages keep the units established by #1391; a genuine idle-counter interval remains zero. Report shapes, enrollment, runtime transport trust-and command authority are unchanged. `TestCollectContainerPodmanCPUIntervals`-in `internal/dockeragent/agent_internal_test.go` exercises decoding through the-real collector, idle/advancing counters, start changes and Docker isolation.-Synthetic controls do not establish a native Podman payload or reporter recovery.+Every one-shot container stats response retains a cumulative CPU baseline.+Podman uses only consecutive counter/read pairs, not its unbound compatibility+`cpu` percentage, even when positive. The nanosecond/wall-clock interval keeps+the 100%-per-core units established by #1391 without Docker's host-core multiplier;+server ingestion alone converts to host capacity. Idle remains zero. First+samples, resets and start changes rebaseline; start changes do so even across a+backward clock. Duplicate or older reads in one lifetime cannot move its baseline+backwards. Report shapes, enrollment, runtime transport trust and command authority+are unchanged. `TestCollectContainerPodmanCPUIntervals` covers these boundaries+through the real collector. `TestPodmanCPUCollectorHistoryAndAlerts` follows the+final compressed report into authenticated ingestion, persistent HTTP History and+alert units on 8- and 4-CPU hosts. Synthetic controls do not establish a native+Podman payload, installed recovery or containing release.### MD RAID required members and spares — issue #2369diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 9f97b1d658..14f3d6003a 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -178,19 +178,29 @@ source selection and wire formats are unchanged. The History cleanup testscover every supported series, byte-array reclamation and connected readers;these controls do not attribute a reporter's native process-memory growth.-### Podman zero-percentage fallback — issue #2447--One-shot compatibility stats with `cpu:0` do not suppress advancing cumulative-CPU samples. The collector records every interval and uses Podman's CPU-time-nanoseconds over elapsed wall-clock nanoseconds, independent of host core count.-A positive finite reported percentage remains authoritative, preserving #1391.-Counter resets or a changed container start establish a new baseline; duplicate-and out-of-order reads cannot create subsequent spikes. No timestamp means no-measurable counter interval. The existing report CPU field, graph/alert units,-Docker system-counter calculation, memory and permission scopes are unchanged.-`TestCollectContainerPodmanCPUIntervals` in `agent_internal_test.go` covers the-actual JSON/collector boundary, including changing percentage availability.-Native Podman shape and graph/alert acceptance remain separate evidence.+### Podman interval CPU selection — issue #2447++Every Podman one-shot response uses consecutive cumulative CPU-time nanoseconds+over elapsed wall-clock nanoseconds. The compatibility `cpu` percentage is not+bound to that interval and cannot replace it, whether positive, zero, missing or+invalid; measured idle remains zero. First samples, counter resets and container+start changes establish a baseline instead of presenting lifetime CPU as current+usage. A new lifetime also rebaselines across a backward clock; duplicate or older+reads in the same lifetime cannot move that baseline. Missing timestamps cannot+invent an interval. The collector keeps the 100%-per-core scale (#1391), and the+server converts exactly once to host-capacity percent for History and alerts.+Docker's system-counter calculation, memory, report shape and permissions stay+unchanged. `Container CPU selected for report` logs the final selected raw value,+not a preliminary calculation or a claim of successful delivery.++`TestCollectContainerPodmanCPUIntervals` covers decoding, idle, restarts, backward+clocks, duplicates and Docker isolation. `TestPodmanCPUCollectorHistoryAndAlerts`+in `internal/dockeragent/cpu_roundtrip_test.go` follows deliberately conflicting+synthetic percentages through real compressed HTTPS report delivery, scoped+authentication, ingestion, memory and persistent HTTP History, and alert identity+and units on 8- and 4-CPU hosts. Genuine high usage still breaches its configured+threshold; no clipping or smoothing masks spikes. These source controls do not+establish the reporter's native payload, installed cause or containing release.### Guest-agent coordination around backups — issue #2439diff --git a/internal/dockeragent/agent_collect_test.go b/internal/dockeragent/agent_collect_test.goindex 74b2f553f4..e111eb44d9 100644--- a/internal/dockeragent/agent_collect_test.go+++ b/internal/dockeragent/agent_collect_test.go@@ -426,11 +426,12 @@ func TestCollectContainer(t *testing.T) {}})- t.Run("podman uses reported cpu percent from compat stats payload", func(t *testing.T) {+ t.Run("podman interval wins over conflicting compat percentage", func(t *testing.T) {+ epoch := time.Date(2026, 4, 9, 12, 0, 0, 0, time.UTC)inspect := baseInspect()inspect.State = &containertypes.State{Running: true,- StartedAt: time.Now().Add(-time.Minute).Format(time.RFC3339Nano),+ StartedAt: epoch.Add(-time.Minute).Format(time.RFC3339Nano),}agent := &Agent{@@ -441,7 +442,8 @@ func TestCollectContainer(t *testing.T) {totalUsage: 100_000_000,systemUsage: 1_000_000_000,onlineCPUs: 16,- read: time.Now().Add(-time.Second),+ read: epoch.Add(-time.Second),+ startedAt: epoch.Add(-time.Minute),},},docker: &fakeDockerClient{@@ -479,8 +481,8 @@ func TestCollectContainer(t *testing.T) {if err != nil {t.Fatalf("unexpected error: %v", err)}- if container.CPUPercent != 0.32 {- t.Fatalf("expected podman cpu percent 0.32 from payload, got %f", container.CPUPercent)+ if container.CPUPercent != 3 {+ t.Fatalf("expected Podman interval CPU 3%% despite cpu:0.32, got %f", container.CPUPercent)}})diff --git a/internal/dockeragent/agent_cpu_test.go b/internal/dockeragent/agent_cpu_test.goindex 3bbfefee98..907ca05d98 100644--- a/internal/dockeragent/agent_cpu_test.go+++ b/internal/dockeragent/agent_cpu_test.go@@ -356,7 +356,7 @@ func TestCalculateContainerCPUPercent(t *testing.T) {})}-func TestDecodeContainerStatsPayloadExtractsPodmanCPU(t *testing.T) {+func TestDecodeContainerStatsPayloadKeepsPodmanIntervalCounters(t *testing.T) {payload := []byte(`{"read":"2026-04-09T12:00:00Z","cpu_stats":{@@ -371,17 +371,14 @@ func TestDecodeContainerStatsPayloadExtractsPodmanCPU(t *testing.T) {"blkio_stats":{"io_service_bytes_recursive":[]}}`)- stats, podmanCPU, err := decodeContainerStatsPayload(payload)+ stats, err := decodeContainerStatsPayload(payload)if err != nil {t.Fatalf("unexpected decode error: %v", err)}if stats.CPUStats.CPUUsage.TotalUsage != 123456789 {t.Fatalf("expected total usage 123456789, got %d", stats.CPUStats.CPUUsage.TotalUsage)}- if podmanCPU == nil {- t.Fatal("expected podman cpu percent to be extracted")- }- if math.Abs(*podmanCPU-0.32) > 0.000001 {- t.Fatalf("expected podman cpu percent 0.32, got %f", *podmanCPU)+ if !stats.Read.Equal(time.Date(2026, 4, 9, 12, 0, 0, 0, time.UTC)) || stats.CPUStats.OnlineCPUs != 16 || stats.MemoryStats.Usage != 1000 {+ t.Fatalf("compat percentage changed counter timestamp, host capacity or memory: %+v", stats)}}diff --git a/internal/dockeragent/agent_internal_test.go b/internal/dockeragent/agent_internal_test.goindex c4bffdc700..e7e034f91a 100644--- a/internal/dockeragent/agent_internal_test.go+++ b/internal/dockeragent/agent_internal_test.go@@ -1872,15 +1872,15 @@ func TestCollectContainerPodmanCPUIntervals(t *testing.T) {{second: 0, total: 100_000_000, percent: "null"},{second: 1, total: 130_000_000, percent: "null", want: 3},}},- {"positive percentages keep units and retain the baseline", RuntimePodman, 16, []sample{- {second: 0, total: 100_000_000, percent: "0.32", want: 0.32},- {second: 1, total: 130_000_000, percent: "0.64", want: 0.64},+ {"positive percentages cannot replace interval evidence", RuntimePodman, 16, []sample{+ {second: 0, total: 100_000_000, percent: "0.32"},+ {second: 1, total: 130_000_000, percent: "0.64", want: 3},{second: 2, total: 160_000_000, percent: "0", want: 3},{second: 3, total: 190_000_000, want: 3},}},{"absent then reported then zero", RuntimePodman, 16, []sample{{second: 0, total: 100_000_000},- {second: 1, total: 130_000_000, percent: "0.32", want: 0.32},+ {second: 1, total: 130_000_000, percent: "0.32", want: 3},{second: 2, total: 160_000_000, percent: "0", want: 3},}},{"invalid negative percentage", RuntimePodman, 16, []sample{@@ -1893,28 +1893,38 @@ func TestCollectContainerPodmanCPUIntervals(t *testing.T) {}},{"counter reset rebaselines rather than using new lifetime", RuntimePodman, 16, []sample{{second: 0, total: 100_000_000, percent: "0"},- {second: 1, total: 10_000_000, percent: "0"},+ {second: 1, total: 10_000_000, percent: "800"},{second: 2, total: 40_000_000, percent: "0", want: 3},}},{"restart with an already larger counter", RuntimePodman, 16, []sample{{second: 0, total: 100_000_000, percent: "0"},- {second: 1, total: 260_000_000, percent: "0", start: 1},+ {second: 1, total: 260_000_000, percent: "800", start: 1},{second: 2, total: 290_000_000, percent: "0", start: 1, want: 3},}},{"repeated and old timestamps do not move the baseline", RuntimePodman, 16, []sample{{second: 0, total: 100_000_000, percent: "0"},{second: 1, total: 130_000_000, percent: "0", want: 3},- {second: 1, total: 150_000_000, percent: "0"},- {second: 0, total: 170_000_000, percent: "0"},+ {second: 1, total: 150_000_000, percent: "800"},+ {second: 0, total: 170_000_000, percent: "800"},{second: 2, total: 160_000_000, percent: "0", want: 3},}},+ {"restart establishes a baseline across a backward clock", RuntimePodman, 8, []sample{+ {second: 3, total: 100_000_000, percent: "0"},+ {second: 2, total: 260_000_000, percent: "800", start: 1},+ {second: 3, total: 290_000_000, percent: "800", start: 1, want: 3},+ }},+ {"positive percentage cannot replace measured idle", RuntimePodman, 4, []sample{+ {second: 0, total: 100_000_000, percent: "25"},+ {second: 1, total: 100_000_000, percent: "800"},+ {second: 2, total: 600_000_000, percent: "92.8", want: 50},+ }},{"missing timestamp cannot invent an interval", RuntimePodman, 16, []sample{- {second: -1, total: 100_000_000, percent: "0"},- {second: 1, total: 130_000_000, percent: "0"},+ {second: -1, total: 100_000_000, percent: "800"},+ {second: 1, total: 130_000_000, percent: "800"},{second: 2, total: 160_000_000, percent: "0", want: 3},}},{"stop clears the baseline", RuntimePodman, 16, []sample{- {second: 0, total: 100_000_000, percent: "0.32", want: 0.32},+ {second: 0, total: 100_000_000, percent: "0.32"},{second: 1, stopped: true},{second: 2, total: 260_000_000, percent: "0"},{second: 3, total: 290_000_000, percent: "0", want: 3},diff --git a/internal/dockeragent/collect.go b/internal/dockeragent/collect.goindex 9ca434c9f8..5c852e02b7 100644--- a/internal/dockeragent/collect.go+++ b/internal/dockeragent/collect.go@@ -596,20 +596,17 @@ func (a *Agent) collectContainer(ctx context.Context, summary containertypes.Sumreturn agentsdocker.Container{}, fmt.Errorf("read stats: %w", err)}- stats, podmanCPUPercent, err := decodeContainerStatsPayload(payload)+ stats, err := decodeContainerStatsPayload(payload)if err != nil {return agentsdocker.Container{}, fmt.Errorf("decode stats: %w", err)}- // Keep a baseline even when Podman supplies a percentage: one-shot- // compatibility responses can report cpu:0 while total_usage advances.- // Preserve positive native percentages (and their units, #1391), but- // use our interval delta for an absent, zero or invalid percentage.+ // Podman's compatibility cpu percentage is not tied to the interval+ // between our one-shot reads. Use consecutive counter/read pairs for+ // every Podman sample, including idle and rebaseline samples. The result+ // stays on the 100%-per-core scale (#1391), as for Docker; only the server+ // converts it into a share of host capacity.cpuPercent = a.calculateContainerCPUPercent(summary.ID, stats, parseTime(inspect.State.StartedAt))- if a.runtime == RuntimePodman && podmanCPUPercent != nil &&- *podmanCPUPercent > 0 && !math.IsInf(*podmanCPUPercent, 0) {- cpuPercent = *podmanCPUPercent- }memUsage, memLimit, memPercent = calculateMemoryUsage(stats)blockIO = summarizeBlockIO(stats)networkRX, networkTX = summarizeNetworkIO(stats)@@ -618,6 +615,11 @@ func (a *Agent) collectContainer(ctx context.Context, summary containertypes.Sumdelete(a.prevContainerCPU, summary.ID)a.cpuMu.Unlock()}+ a.logger.Debug().+ Str("container_id", summary.ID).+ Str("runtime", string(a.runtime)).+ Float64("cpu_percent", cpuPercent).+ Msg("Container CPU selected for report")createdAt := time.Unix(summary.Created, 0)@@ -1122,24 +1124,12 @@ func extractPodmanMetadata(labels map[string]string) *agentsdocker.PodmanContainreturn meta}-type podmanCompatStatsProbe struct {- CPUStats struct {- CPU *float64 `json:"cpu"`- } `json:"cpu_stats"`-}--func decodeContainerStatsPayload(payload []byte) (containertypes.StatsResponse, *float64, error) {+func decodeContainerStatsPayload(payload []byte) (containertypes.StatsResponse, error) {var stats containertypes.StatsResponseif err := json.Unmarshal(payload, &stats); err != nil {- return containertypes.StatsResponse{}, nil, err- }-- var probe podmanCompatStatsProbe- if err := json.Unmarshal(payload, &probe); err != nil {- return stats, nil, nil+ return containertypes.StatsResponse{}, err}-- return stats, probe.CPUStats.CPU, nil+ return stats, nil}func (a *Agent) calculateContainerCPUPercent(containerID string, stats containertypes.StatsResponse, startedAt time.Time) float64 {@@ -1160,6 +1150,12 @@ func (a *Agent) calculateContainerCPUPercent(containerID string, stats container// (from container start) and producing a constant lifetime-average CPU%// instead of a current value.prev, ok := a.prevContainerCPU[containerID]+ if a.runtime == RuntimePodman && ok && !prev.startedAt.IsZero() && !current.startedAt.IsZero() && !current.startedAt.Equal(prev.startedAt) {+ // A new lifetime must establish its own baseline even if the runtime's+ // clock has moved back. Timestamp ordering only compares one lifetime.+ a.prevContainerCPU[containerID] = current+ return 0+ }if a.runtime == RuntimePodman && ok && !prev.read.IsZero() && !current.read.IsZero() && !current.read.After(prev.read) {// Repeated or out-of-order responses must not move the baseline back// and manufacture usage in the next interval.@@ -1182,10 +1178,8 @@ func (a *Agent) calculateContainerCPUPercent(containerID string, stats container// We have a previous sample - update it after calculationa.prevContainerCPU[containerID] = current- if a.runtime == RuntimePodman && (current.totalUsage < prev.totalUsage ||- (!prev.startedAt.IsZero() && !current.startedAt.IsZero() && !current.startedAt.Equal(prev.startedAt))) {- // A restart is not an interval sample, even if the new lifetime counter- // has already exceeded the old one. Start again from this baseline.+ if a.runtime == RuntimePodman && current.totalUsage < prev.totalUsage {+ // A counter reset is not an interval sample. Start again from this baseline.return 0}