Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep Podman CPU reports on their measured interval

pulse-triage[bot]0c88b83fed81
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 8311a06f92..51a31e9ad2 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -59,18 +59,19 @@ system-information probe; no synthetic success establishes installed acceptance.
### Podman interval CPU evidence — issue #2447
-Every one-shot container stats response retains a cumulative CPU baseline,
-even when a positive Podman compatibility percentage takes precedence. Zero,
-absent or invalid percentages use the valid nanosecond/wall-clock interval,
-without Docker's host-core multiplier. First samples, counter resets and changes
-of container start time rebaseline rather than presenting lifetime CPU as current
-usage. Repeated or older read timestamps cannot move the baseline backwards.
-Positive native percentages keep the units established by #1391; a genuine idle
-counter interval remains zero. Report shapes, enrollment, runtime transport trust
-and command authority are unchanged. `TestCollectContainerPodmanCPUIntervals`
-in `internal/dockeragent/agent_internal_test.go` exercises decoding through the
-real collector, idle/advancing counters, start changes and Docker isolation.
-Synthetic controls do not establish a native Podman payload or reporter recovery.
+Every one-shot container stats response retains a cumulative CPU baseline.
+Podman uses only consecutive counter/read pairs, not its unbound compatibility
+`cpu` percentage, even when positive. The nanosecond/wall-clock interval keeps
+the 100%-per-core units established by #1391 without Docker's host-core multiplier;
+server ingestion alone converts to host capacity. Idle remains zero. First
+samples, resets and start changes rebaseline; start changes do so even across a
+backward clock. Duplicate or older reads in one lifetime cannot move its baseline
+backwards. Report shapes, enrollment, runtime transport trust and command authority
+are unchanged. `TestCollectContainerPodmanCPUIntervals` covers these boundaries
+through the real collector. `TestPodmanCPUCollectorHistoryAndAlerts` follows the
+final compressed report into authenticated ingestion, persistent HTTP History and
+alert units on 8- and 4-CPU hosts. Synthetic controls do not establish a native
+Podman payload, installed recovery or containing release.
### MD RAID required members and spares — issue #2369
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 9f97b1d658..14f3d6003a 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -178,19 +178,29 @@ source selection and wire formats are unchanged. The History cleanup tests
cover every supported series, byte-array reclamation and connected readers;
these controls do not attribute a reporter's native process-memory growth.
-### Podman zero-percentage fallback — issue #2447
-
-One-shot compatibility stats with `cpu:0` do not suppress advancing cumulative
-CPU samples. The collector records every interval and uses Podman's CPU-time
-nanoseconds over elapsed wall-clock nanoseconds, independent of host core count.
-A positive finite reported percentage remains authoritative, preserving #1391.
-Counter resets or a changed container start establish a new baseline; duplicate
-and out-of-order reads cannot create subsequent spikes. No timestamp means no
-measurable counter interval. The existing report CPU field, graph/alert units,
-Docker system-counter calculation, memory and permission scopes are unchanged.
-`TestCollectContainerPodmanCPUIntervals` in `agent_internal_test.go` covers the
-actual JSON/collector boundary, including changing percentage availability.
-Native Podman shape and graph/alert acceptance remain separate evidence.
+### Podman interval CPU selection — issue #2447
+
+Every Podman one-shot response uses consecutive cumulative CPU-time nanoseconds
+over elapsed wall-clock nanoseconds. The compatibility `cpu` percentage is not
+bound to that interval and cannot replace it, whether positive, zero, missing or
+invalid; measured idle remains zero. First samples, counter resets and container
+start changes establish a baseline instead of presenting lifetime CPU as current
+usage. A new lifetime also rebaselines across a backward clock; duplicate or older
+reads in the same lifetime cannot move that baseline. Missing timestamps cannot
+invent an interval. The collector keeps the 100%-per-core scale (#1391), and the
+server converts exactly once to host-capacity percent for History and alerts.
+Docker's system-counter calculation, memory, report shape and permissions stay
+unchanged. `Container CPU selected for report` logs the final selected raw value,
+not a preliminary calculation or a claim of successful delivery.
+
+`TestCollectContainerPodmanCPUIntervals` covers decoding, idle, restarts, backward
+clocks, duplicates and Docker isolation. `TestPodmanCPUCollectorHistoryAndAlerts`
+in `internal/dockeragent/cpu_roundtrip_test.go` follows deliberately conflicting
+synthetic percentages through real compressed HTTPS report delivery, scoped
+authentication, ingestion, memory and persistent HTTP History, and alert identity
+and units on 8- and 4-CPU hosts. Genuine high usage still breaches its configured
+threshold; no clipping or smoothing masks spikes. These source controls do not
+establish the reporter's native payload, installed cause or containing release.
### Guest-agent coordination around backups — issue #2439
diff --git a/internal/dockeragent/agent_collect_test.go b/internal/dockeragent/agent_collect_test.go
index 74b2f553f4..e111eb44d9 100644
--- a/internal/dockeragent/agent_collect_test.go
+++ b/internal/dockeragent/agent_collect_test.go
@@ -426,11 +426,12 @@ func TestCollectContainer(t *testing.T) {
}
})
- t.Run("podman uses reported cpu percent from compat stats payload", func(t *testing.T) {
+ t.Run("podman interval wins over conflicting compat percentage", func(t *testing.T) {
+ epoch := time.Date(2026, 4, 9, 12, 0, 0, 0, time.UTC)
inspect := baseInspect()
inspect.State = &containertypes.State{
Running: true,
- StartedAt: time.Now().Add(-time.Minute).Format(time.RFC3339Nano),
+ StartedAt: epoch.Add(-time.Minute).Format(time.RFC3339Nano),
}
agent := &Agent{
@@ -441,7 +442,8 @@ func TestCollectContainer(t *testing.T) {
totalUsage: 100_000_000,
systemUsage: 1_000_000_000,
onlineCPUs: 16,
- read: time.Now().Add(-time.Second),
+ read: epoch.Add(-time.Second),
+ startedAt: epoch.Add(-time.Minute),
},
},
docker: &fakeDockerClient{
@@ -479,8 +481,8 @@ func TestCollectContainer(t *testing.T) {
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
- if container.CPUPercent != 0.32 {
- t.Fatalf("expected podman cpu percent 0.32 from payload, got %f", container.CPUPercent)
+ if container.CPUPercent != 3 {
+ t.Fatalf("expected Podman interval CPU 3%% despite cpu:0.32, got %f", container.CPUPercent)
}
})
diff --git a/internal/dockeragent/agent_cpu_test.go b/internal/dockeragent/agent_cpu_test.go
index 3bbfefee98..907ca05d98 100644
--- a/internal/dockeragent/agent_cpu_test.go
+++ b/internal/dockeragent/agent_cpu_test.go
@@ -356,7 +356,7 @@ func TestCalculateContainerCPUPercent(t *testing.T) {
})
}
-func TestDecodeContainerStatsPayloadExtractsPodmanCPU(t *testing.T) {
+func TestDecodeContainerStatsPayloadKeepsPodmanIntervalCounters(t *testing.T) {
payload := []byte(`{
"read":"2026-04-09T12:00:00Z",
"cpu_stats":{
@@ -371,17 +371,14 @@ func TestDecodeContainerStatsPayloadExtractsPodmanCPU(t *testing.T) {
"blkio_stats":{"io_service_bytes_recursive":[]}
}`)
- stats, podmanCPU, err := decodeContainerStatsPayload(payload)
+ stats, err := decodeContainerStatsPayload(payload)
if err != nil {
t.Fatalf("unexpected decode error: %v", err)
}
if stats.CPUStats.CPUUsage.TotalUsage != 123456789 {
t.Fatalf("expected total usage 123456789, got %d", stats.CPUStats.CPUUsage.TotalUsage)
}
- if podmanCPU == nil {
- t.Fatal("expected podman cpu percent to be extracted")
- }
- if math.Abs(*podmanCPU-0.32) > 0.000001 {
- t.Fatalf("expected podman cpu percent 0.32, got %f", *podmanCPU)
+ if !stats.Read.Equal(time.Date(2026, 4, 9, 12, 0, 0, 0, time.UTC)) || stats.CPUStats.OnlineCPUs != 16 || stats.MemoryStats.Usage != 1000 {
+ t.Fatalf("compat percentage changed counter timestamp, host capacity or memory: %+v", stats)
}
}
diff --git a/internal/dockeragent/agent_internal_test.go b/internal/dockeragent/agent_internal_test.go
index c4bffdc700..e7e034f91a 100644
--- a/internal/dockeragent/agent_internal_test.go
+++ b/internal/dockeragent/agent_internal_test.go
@@ -1872,15 +1872,15 @@ func TestCollectContainerPodmanCPUIntervals(t *testing.T) {
{second: 0, total: 100_000_000, percent: "null"},
{second: 1, total: 130_000_000, percent: "null", want: 3},
}},
- {"positive percentages keep units and retain the baseline", RuntimePodman, 16, []sample{
- {second: 0, total: 100_000_000, percent: "0.32", want: 0.32},
- {second: 1, total: 130_000_000, percent: "0.64", want: 0.64},
+ {"positive percentages cannot replace interval evidence", RuntimePodman, 16, []sample{
+ {second: 0, total: 100_000_000, percent: "0.32"},
+ {second: 1, total: 130_000_000, percent: "0.64", want: 3},
{second: 2, total: 160_000_000, percent: "0", want: 3},
{second: 3, total: 190_000_000, want: 3},
}},
{"absent then reported then zero", RuntimePodman, 16, []sample{
{second: 0, total: 100_000_000},
- {second: 1, total: 130_000_000, percent: "0.32", want: 0.32},
+ {second: 1, total: 130_000_000, percent: "0.32", want: 3},
{second: 2, total: 160_000_000, percent: "0", want: 3},
}},
{"invalid negative percentage", RuntimePodman, 16, []sample{
@@ -1893,28 +1893,38 @@ func TestCollectContainerPodmanCPUIntervals(t *testing.T) {
}},
{"counter reset rebaselines rather than using new lifetime", RuntimePodman, 16, []sample{
{second: 0, total: 100_000_000, percent: "0"},
- {second: 1, total: 10_000_000, percent: "0"},
+ {second: 1, total: 10_000_000, percent: "800"},
{second: 2, total: 40_000_000, percent: "0", want: 3},
}},
{"restart with an already larger counter", RuntimePodman, 16, []sample{
{second: 0, total: 100_000_000, percent: "0"},
- {second: 1, total: 260_000_000, percent: "0", start: 1},
+ {second: 1, total: 260_000_000, percent: "800", start: 1},
{second: 2, total: 290_000_000, percent: "0", start: 1, want: 3},
}},
{"repeated and old timestamps do not move the baseline", RuntimePodman, 16, []sample{
{second: 0, total: 100_000_000, percent: "0"},
{second: 1, total: 130_000_000, percent: "0", want: 3},
- {second: 1, total: 150_000_000, percent: "0"},
- {second: 0, total: 170_000_000, percent: "0"},
+ {second: 1, total: 150_000_000, percent: "800"},
+ {second: 0, total: 170_000_000, percent: "800"},
{second: 2, total: 160_000_000, percent: "0", want: 3},
}},
+ {"restart establishes a baseline across a backward clock", RuntimePodman, 8, []sample{
+ {second: 3, total: 100_000_000, percent: "0"},
+ {second: 2, total: 260_000_000, percent: "800", start: 1},
+ {second: 3, total: 290_000_000, percent: "800", start: 1, want: 3},
+ }},
+ {"positive percentage cannot replace measured idle", RuntimePodman, 4, []sample{
+ {second: 0, total: 100_000_000, percent: "25"},
+ {second: 1, total: 100_000_000, percent: "800"},
+ {second: 2, total: 600_000_000, percent: "92.8", want: 50},
+ }},
{"missing timestamp cannot invent an interval", RuntimePodman, 16, []sample{
- {second: -1, total: 100_000_000, percent: "0"},
- {second: 1, total: 130_000_000, percent: "0"},
+ {second: -1, total: 100_000_000, percent: "800"},
+ {second: 1, total: 130_000_000, percent: "800"},
{second: 2, total: 160_000_000, percent: "0", want: 3},
}},
{"stop clears the baseline", RuntimePodman, 16, []sample{
- {second: 0, total: 100_000_000, percent: "0.32", want: 0.32},
+ {second: 0, total: 100_000_000, percent: "0.32"},
{second: 1, stopped: true},
{second: 2, total: 260_000_000, percent: "0"},
{second: 3, total: 290_000_000, percent: "0", want: 3},
diff --git a/internal/dockeragent/collect.go b/internal/dockeragent/collect.go
index 9ca434c9f8..5c852e02b7 100644
--- a/internal/dockeragent/collect.go
+++ b/internal/dockeragent/collect.go
@@ -596,20 +596,17 @@ func (a *Agent) collectContainer(ctx context.Context, summary containertypes.Sum
return agentsdocker.Container{}, fmt.Errorf("read stats: %w", err)
}
- stats, podmanCPUPercent, err := decodeContainerStatsPayload(payload)
+ stats, err := decodeContainerStatsPayload(payload)
if err != nil {
return agentsdocker.Container{}, fmt.Errorf("decode stats: %w", err)
}
- // Keep a baseline even when Podman supplies a percentage: one-shot
- // compatibility responses can report cpu:0 while total_usage advances.
- // Preserve positive native percentages (and their units, #1391), but
- // use our interval delta for an absent, zero or invalid percentage.
+ // Podman's compatibility cpu percentage is not tied to the interval
+ // between our one-shot reads. Use consecutive counter/read pairs for
+ // every Podman sample, including idle and rebaseline samples. The result
+ // stays on the 100%-per-core scale (#1391), as for Docker; only the server
+ // converts it into a share of host capacity.
cpuPercent = a.calculateContainerCPUPercent(summary.ID, stats, parseTime(inspect.State.StartedAt))
- if a.runtime == RuntimePodman && podmanCPUPercent != nil &&
- *podmanCPUPercent > 0 && !math.IsInf(*podmanCPUPercent, 0) {
- cpuPercent = *podmanCPUPercent
- }
memUsage, memLimit, memPercent = calculateMemoryUsage(stats)
blockIO = summarizeBlockIO(stats)
networkRX, networkTX = summarizeNetworkIO(stats)
@@ -618,6 +615,11 @@ func (a *Agent) collectContainer(ctx context.Context, summary containertypes.Sum
delete(a.prevContainerCPU, summary.ID)
a.cpuMu.Unlock()
}
+ a.logger.Debug().
+ Str("container_id", summary.ID).
+ Str("runtime", string(a.runtime)).
+ Float64("cpu_percent", cpuPercent).
+ Msg("Container CPU selected for report")
createdAt := time.Unix(summary.Created, 0)
@@ -1122,24 +1124,12 @@ func extractPodmanMetadata(labels map[string]string) *agentsdocker.PodmanContain
return meta
}
-type podmanCompatStatsProbe struct {
- CPUStats struct {
- CPU *float64 `json:"cpu"`
- } `json:"cpu_stats"`
-}
-
-func decodeContainerStatsPayload(payload []byte) (containertypes.StatsResponse, *float64, error) {
+func decodeContainerStatsPayload(payload []byte) (containertypes.StatsResponse, error) {
var stats containertypes.StatsResponse
if err := json.Unmarshal(payload, &stats); err != nil {
- return containertypes.StatsResponse{}, nil, err
- }
-
- var probe podmanCompatStatsProbe
- if err := json.Unmarshal(payload, &probe); err != nil {
- return stats, nil, nil
+ return containertypes.StatsResponse{}, err
}
-
- return stats, probe.CPUStats.CPU, nil
+ return stats, nil
}
func (a *Agent) calculateContainerCPUPercent(containerID string, stats containertypes.StatsResponse, startedAt time.Time) float64 {
@@ -1160,6 +1150,12 @@ func (a *Agent) calculateContainerCPUPercent(containerID string, stats container
// (from container start) and producing a constant lifetime-average CPU%
// instead of a current value.
prev, ok := a.prevContainerCPU[containerID]
+ if a.runtime == RuntimePodman && ok && !prev.startedAt.IsZero() && !current.startedAt.IsZero() && !current.startedAt.Equal(prev.startedAt) {
+ // A new lifetime must establish its own baseline even if the runtime's
+ // clock has moved back. Timestamp ordering only compares one lifetime.
+ a.prevContainerCPU[containerID] = current
+ return 0
+ }
if a.runtime == RuntimePodman && ok && !prev.read.IsZero() && !current.read.IsZero() && !current.read.After(prev.read) {
// Repeated or out-of-order responses must not move the baseline back
// and manufacture usage in the next interval.
@@ -1182,10 +1178,8 @@ func (a *Agent) calculateContainerCPUPercent(containerID string, stats container
// We have a previous sample - update it after calculation
a.prevContainerCPU[containerID] = current
- if a.runtime == RuntimePodman && (current.totalUsage < prev.totalUsage ||
- (!prev.startedAt.IsZero() && !current.startedAt.IsZero() && !current.startedAt.Equal(prev.startedAt))) {
- // A restart is not an interval sample, even if the new lifetime counter
- // has already exceeded the old one. Start again from this baseline.
+ if a.runtime == RuntimePodman && current.totalUsage < prev.totalUsage {
+ // A counter reset is not an interval sample. Start again from this baseline.
return 0
}