Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

fix(monitoring): use collection clocks for agent counter rates

rcourtman6dd202276c1d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 0a3932108e..653ee593a4 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -2341,7 +2341,10 @@ update, profile rollout, command reachability, or fleet-control authority.
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
Every current agent process authors a process-unique report stream and a
- monotonic sequence within that stream. Reconnect delivery must preserve
+ monotonic sequence within that stream. The optional
+ `ioCountersCollectedAt` records when the metrics snapshot read the
+ cumulative disk-I/O and network counters; `timestamp` stays the report's
+ end-of-collection time for age and order. Reconnect delivery must preserve
collection order by draining each destination's persisted FIFO before
sending a newly collected report, so buffered running-task snapshots cannot
overwrite a later cancellation or completion. Authentication/scope
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index c0897f0364..e9fcb7a166 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -3505,6 +3505,34 @@ buffered complete inventory overwrite a newer helper-degraded status report.
Legacy reports without a sequence retain timestamp-based reconnect-burst
protection, but a clock correction after a normal report interval must be
admitted rather than freezing telemetry indefinitely.
+Counter rates follow the same ownership. An accepted sequenced report that
+carries its authored time is measured on the reporting machine's clock: host
+network and disk, physical-disk and container rates divide by the interval
+between the authored collection times of consecutive accepted reports, across
+agent restarts too, so a FIFO drained back to back cannot compress collection
+intervals into milliseconds of receipt time and fabricate throughput spikes.
+A current host agent also sends `ioCountersCollectedAt`, stamped right after
+the metrics snapshot reads its disk-I/O and network counters. Host and
+physical-disk rates are measured on it rather than on the report `timestamp`,
+which follows every optional collector, and a switch between the two sampling
+modes starts a new counter baseline.
+Libvirt domain CPU and I/O counters inside an accepted host report are measured
+on the inventory's `CollectedAt`, which the agent stamps on the same clock just
+before it reads the domain stats, and CPU usage shares the I/O interval rather
+than the gap since the previous host report.
+Legacy and untimestamped reports keep receipt-time rates. A switch between the
+two clocks starts a new counter baseline, and an authored time that does not
+advance past the baseline, a backward step or a repeated timestamp, drops it,
+so no rate is reported until two samples bound a new interval. Liveness,
+history timestamps and rate-series cleanup keep Pulse receipt time.
+Unknown Docker network intervals must be omitted from both history writers;
+the numeric compatibility fields must not turn first samples or clock faults
+into measured idle. A genuine unchanged-counter interval still writes zero.
+Libvirt CPU counter baselines retain the full unsigned nanosecond range rather
+than clamping accumulated CPU time to a signed byte-counter value. The
+`agent_report_counter_clock_test.go`, `agent_report_io_counter_clock_test.go`
+and `agent_report_libvirt_clock_test.go` regressions cover reconnect delivery,
+clock transitions, idle/reset intervals and counter range preservation.
Host telemetry also has a reporting lease derived from the agent cadence.
Monitoring must not clear a genuinely active Unraid parity operation or Linux
RAID rebuild during an ordinary polling gap. Once the lease expires, it must
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index 00863a1873..5c60f4a468 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -1547,9 +1547,13 @@
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
"internal/config/host_continuity_test.go",
"internal/models/metrics_types_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/availability_probe_agent_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
@@ -1654,7 +1658,8 @@
"test_prefixes": [],
"exact_files": [
"internal/hostmetrics/collector_stuck_mount_test.go",
- "internal/hostmetrics/collector_test.go"
+ "internal/hostmetrics/collector_test.go",
+ "internal/hostmetrics/io_counters_collected_at_test.go"
]
},
{
@@ -1670,6 +1675,7 @@
"test_prefixes": [],
"exact_files": [
"internal/agentupdate/coverage_test.go",
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/hostagent/action_runner_client_test.go",
"internal/hostagent/agent_flushbuffer_test.go",
"internal/hostagent/agent_metrics_test.go",
@@ -1689,6 +1695,7 @@
"internal/hostagent/smartctl_standby_guard_test.go",
"internal/hostagent/storage_cleanup_test.go",
"internal/hostagent/unraid_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go"
]
},
@@ -6590,6 +6597,9 @@
"test_prefixes": [],
"exact_files": [
"internal/config/host_continuity_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
"internal/monitoring/issue1485_unraid_lifecycle_test.go",
@@ -6599,6 +6609,7 @@
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_package_updates_test.go",
"internal/unifiedresources/code_standards_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go"
]
},
@@ -7060,6 +7071,9 @@
"internal/config/docker_metadata_test.go",
"internal/config/guest_metadata_test.go",
"internal/config/guest_metadata_writer_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/availability_poller_test.go",
"internal/monitoring/availability_probe_agent_test.go",
"internal/monitoring/availability_udp_test.go",
diff --git a/internal/api/agent_report_decode_io_stamp_compat_test.go b/internal/api/agent_report_decode_io_stamp_compat_test.go
new file mode 100644
index 0000000000..be16110f02
--- /dev/null
+++ b/internal/api/agent_report_decode_io_stamp_compat_test.go
@@ -0,0 +1,91 @@
+package api
+
+import (
+ "bytes"
+ "compress/gzip"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+)
+
+// preIOStampHostReport is how a receiver built before ioCountersCollectedAt
+// existed sees a host report: the identity, ordering and counter fields it
+// already relies on, and no counter read stamp.
+type preIOStampHostReport struct {
+ Agent struct {
+ ID string `json:"id"`
+ IntervalSeconds int `json:"intervalSeconds"`
+ } `json:"agent"`
+ Host struct {
+ ID string `json:"id"`
+ Hostname string `json:"hostname"`
+ } `json:"host"`
+ Network []struct {
+ Name string `json:"name"`
+ RXBytes uint64 `json:"rxBytes"`
+ } `json:"network"`
+ DiskIO []struct {
+ Device string `json:"device"`
+ ReadBytes uint64 `json:"readBytes"`
+ } `json:"diskIO"`
+ Timestamp time.Time `json:"timestamp"`
+ SequenceID string `json:"sequenceId"`
+}
+
+const ioStampedHostReportJSON = `{"agent":{"id":"agent-a","intervalSeconds":30},` +
+ `"host":{"id":"host-a","hostname":"host-a"},` +
+ `"network":[{"name":"eth0","rxBytes":1000}],"diskIO":[{"device":"sda","readBytes":2000}],` +
+ `"timestamp":"2026-10-30T12:00:50Z","sequenceId":"v1:stream-a:2",` +
+ `"ioCountersCollectedAt":"2026-10-30T12:00:30.123456789Z"}`
+
+// Preservation check: the report decoder accepts unknown fields, so a
+// receiver without the optional counter read stamp still decodes a stamped
+// report, plain or gzip, keeps every field it knows and ignores the stamp.
+// It passes before and after the stamp is added.
+func TestDecodeCompressedAgentReportIgnoresIOCounterStampOnPreChangeReceiver(t *testing.T) {
+ var compressed bytes.Buffer
+ writer := gzip.NewWriter(&compressed)
+ if _, err := writer.Write([]byte(ioStampedHostReportJSON)); err != nil {
+ t.Fatalf("gzip report: %v", err)
+ }
+ if err := writer.Close(); err != nil {
+ t.Fatalf("close gzip writer: %v", err)
+ }
+
+ for _, tc := range []struct {
+ name string
+ body []byte
+ encoding string
+ }{
+ {"uncompressed", []byte(ioStampedHostReportJSON), ""},
+ {"gzip", compressed.Bytes(), "gzip"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ request := httptest.NewRequest(http.MethodPost, "/report", bytes.NewReader(tc.body))
+ request.Header.Set("Content-Type", "application/json")
+ if tc.encoding != "" {
+ request.Header.Set("Content-Encoding", tc.encoding)
+ }
+ recorder := httptest.NewRecorder()
+
+ var report preIOStampHostReport
+ if !decodeCompressedAgentReport(recorder, request, 256*1024, 1536*1024, &report) {
+ t.Fatalf("decoder refused a report carrying the optional stamp: %d %s", recorder.Code, recorder.Body.String())
+ }
+ if recorder.Body.Len() != 0 {
+ t.Fatalf("decoder wrote a response on success: %s", recorder.Body.String())
+ }
+ wantTimestamp := time.Date(2026, 10, 30, 12, 0, 50, 0, time.UTC)
+ if report.Agent.ID != "agent-a" || report.Agent.IntervalSeconds != 30 ||
+ report.Host.ID != "host-a" || report.Host.Hostname != "host-a" ||
+ !report.Timestamp.Equal(wantTimestamp) || report.SequenceID != "v1:stream-a:2" {
+ t.Fatalf("known fields lost: %+v", report)
+ }
+ if len(report.Network) != 1 || report.Network[0].Name != "eth0" || report.Network[0].RXBytes != 1000 ||
+ len(report.DiskIO) != 1 || report.DiskIO[0].Device != "sda" || report.DiskIO[0].ReadBytes != 2000 {
+ t.Fatalf("counters lost: network %+v disk %+v", report.Network, report.DiskIO)
+ }
+ })
+ }
+}
diff --git a/internal/hostagent/agent.go b/internal/hostagent/agent.go
index a23e76b78b..490d4ee307 100644
--- a/internal/hostagent/agent.go
+++ b/internal/hostagent/agent.go
@@ -1448,6 +1448,9 @@ func (a *Agent) buildReport(ctx context.Context) (agentshost.Report, error) {
Tags: append([]string(nil), runtimeConfig.tags...),
Timestamp: a.collector.Now(),
SequenceID: a.nextReportSequenceID(),
+ // The counters were read inside the metrics snapshot, before the
+ // optional collectors above; Timestamp only marks the report's end.
+ IOCountersCollectedAt: snapshotIOCountersCollectedAt(snapshot),
}
return report, nil
diff --git a/internal/hostagent/agent_metrics_test.go b/internal/hostagent/agent_metrics_test.go
index c457a1701e..de4c1be566 100644
--- a/internal/hostagent/agent_metrics_test.go
+++ b/internal/hostagent/agent_metrics_test.go
@@ -95,7 +95,8 @@ func TestBuildReport(t *testing.T) {
},
metricsFn: func(ctx context.Context, diskExclude []string) (hostmetrics.Snapshot, error) {
return hostmetrics.Snapshot{
- CPUUsagePercent: 50.0,
+ IOCountersCollectedAt: fixedTime.Add(-20 * time.Second),
+ CPUUsagePercent: 50.0,
Memory: agentshost.MemoryMetric{
TotalBytes: 1000,
UsedBytes: 500,
@@ -179,6 +180,11 @@ func TestBuildReport(t *testing.T) {
if !report.Timestamp.Equal(fixedTime) {
t.Errorf("Timestamp = %v, want %v", report.Timestamp, fixedTime)
}
+ // The real report builder must carry the metrics snapshot's read time,
+ // independently of the later end-of-collection timestamp.
+ if report.IOCountersCollectedAt == nil || !report.IOCountersCollectedAt.Equal(fixedTime.Add(-20*time.Second)) {
+ t.Fatalf("IOCountersCollectedAt = %v, want the earlier snapshot read", report.IOCountersCollectedAt)
+ }
})
// Test case 2: Uptime failure
diff --git a/internal/hostagent/sysinfo.go b/internal/hostagent/sysinfo.go
index 261c0c7274..e309c89a94 100644
--- a/internal/hostagent/sysinfo.go
+++ b/internal/hostagent/sysinfo.go
@@ -55,6 +55,16 @@ func collectMetricsWithDiskFilters(ctx context.Context, collector SystemCollecto
return collector.Metrics(ctx, exclude)
}
+// snapshotIOCountersCollectedAt carries the snapshot's counter read time onto
+// the wire, absent when the collector did not stamp one.
+func snapshotIOCountersCollectedAt(snapshot hostmetrics.Snapshot) *time.Time {
+ if snapshot.IOCountersCollectedAt.IsZero() {
+ return nil
+ }
+ collectedAt := snapshot.IOCountersCollectedAt.UTC()
+ return &collectedAt
+}
+
// NewDefaultCollector returns a SystemCollector that uses real OS calls.
func NewDefaultCollector() SystemCollector {
return &defaultCollector{}
diff --git a/internal/hostmetrics/collector.go b/internal/hostmetrics/collector.go
index 4f7ee9c9d6..aa7396cff7 100644
--- a/internal/hostmetrics/collector.go
+++ b/internal/hostmetrics/collector.go
@@ -33,6 +33,7 @@ var (
diskIOCounters = godisk.IOCountersWithContext
netInterfaces = gonet.InterfacesWithContext
netIOCounters = gonet.IOCountersWithContext
+ readZFSARCSize = readARCSize
)
// partitionsVisibleToAgent asks gopsutil to enumerate the mount namespace the
@@ -153,8 +154,15 @@ type Snapshot struct {
Disks []agentshost.Disk
DiskIO []agentshost.DiskIO
Network []agentshost.NetworkInterface
+ // IOCountersCollectedAt is when the cumulative DiskIO and Network counters
+ // were read: stamped right after the network read, which follows the disk
+ // read with only the interface listing between them.
+ IOCountersCollectedAt time.Time
}
+// ioCountersReadNow stamps IOCountersCollectedAt; tests replace it.
+var ioCountersReadNow = func() time.Time { return time.Now().UTC() }
+
// Collector owns the CPU baseline for one reporting loop. Its zero value is
// ready to use. Retain it across reports; do not copy it after first use.
type Collector struct {
@@ -236,7 +244,7 @@ func (c *Collector) CollectWithDiskFilters(ctx context.Context, diskExclude, dis
// ZFS ARC memory is reclaimable under pressure but is counted as "used" by
// both FreeBSD (wired memory) and Linux (not in MemAvailable, openzfs/zfs#10255).
// Subtract it from Used to reflect actual memory pressure. Refs: #1264/#1051
- if arcSize, err := readARCSize(); err == nil && arcSize > 0 {
+ if arcSize, err := readZFSARCSize(); err == nil && arcSize > 0 {
if arcSize < usedBytes {
usedBytes -= arcSize
} else {
@@ -271,6 +279,7 @@ func (c *Collector) CollectWithDiskFilters(ctx context.Context, diskExclude, dis
snapshot.Disks = collectDisksWithIncludes(collectCtx, diskExclude, diskInclude)
snapshot.DiskIO = collectDiskIO(collectCtx, diskExclude)
snapshot.Network = collectNetwork(collectCtx)
+ snapshot.IOCountersCollectedAt = ioCountersReadNow()
return snapshot, nil
}
diff --git a/internal/hostmetrics/io_counters_collected_at_test.go b/internal/hostmetrics/io_counters_collected_at_test.go
new file mode 100644
index 0000000000..438e175b90
--- /dev/null
+++ b/internal/hostmetrics/io_counters_collected_at_test.go
@@ -0,0 +1,78 @@
+package hostmetrics
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ gocpu "github.com/shirou/gopsutil/v4/cpu"
+ godisk "github.com/shirou/gopsutil/v4/disk"
+ goload "github.com/shirou/gopsutil/v4/load"
+ gomem "github.com/shirou/gopsutil/v4/mem"
+ gonet "github.com/shirou/gopsutil/v4/net"
+)
+
+// The counter read stamp is what the server divides counter deltas by, so it
+// must mark the disk and network counter reads rather than the start of the
+// snapshot: earlier steps can stall, here a memory read that takes 20s. Every
+// seam the snapshot reaches is stubbed, so the test touches no host or proc
+// file; it replaces package globals and must stay sequential.
+func TestCollectWithDiskFiltersStampsIOCountersAtTheirRead(t *testing.T) {
+ start := time.Date(2026, 10, 30, 12, 0, 0, 0, time.UTC)
+ clock := start
+ var diskReadAt, netReadAt time.Time
+
+ origCounts, origPercent, origTimes, origLoad, origMemory := cpuCounts, cpuPercent, cpuTimes, loadAvg, virtualMemory
+ origPartitions, origDiskIO, origInterfaces, origNetIO, origNow := diskPartitions, diskIOCounters, netInterfaces, netIOCounters, ioCountersReadNow
+ origARC := readZFSARCSize
+ t.Cleanup(func() {
+ cpuCounts, cpuPercent, cpuTimes, loadAvg, virtualMemory = origCounts, origPercent, origTimes, origLoad, origMemory
+ diskPartitions, diskIOCounters, netInterfaces, netIOCounters, ioCountersReadNow = origPartitions, origDiskIO, origInterfaces, origNetIO, origNow
+ readZFSARCSize = origARC
+ })
+
+ readZFSARCSize = func() (uint64, error) { return 0, nil }
+ cpuCounts = func(context.Context, bool) (int, error) { return 4, nil }
+ cpuPercent = func(context.Context, time.Duration, bool) ([]float64, error) { return []float64{5}, nil }
+ cpuTimes = func(context.Context, bool) ([]gocpu.TimesStat, error) {
+ return []gocpu.TimesStat{{User: 10, Idle: 90}}, nil
+ }
+ loadAvg = func(context.Context) (*goload.AvgStat, error) { return &goload.AvgStat{}, nil }
+ virtualMemory = func(context.Context) (*gomem.VirtualMemoryStat, error) {
+ clock = clock.Add(20 * time.Second)
+ return &gomem.VirtualMemoryStat{Total: 8 << 30, Available: 4 << 30, Free: 2 << 30, Used: 4 << 30}, nil
+ }
+ diskPartitions = func(context.Context, bool) ([]godisk.PartitionStat, error) { return nil, nil }
+ diskIOCounters = func(context.Context, ...string) (map[string]godisk.IOCountersStat, error) {
+ diskReadAt = clock
+ clock = clock.Add(time.Millisecond)
+ return map[string]godisk.IOCountersStat{"sda": {Name: "sda", ReadBytes: 1 << 20, WriteBytes: 2 << 20}}, nil
+ }
+ netInterfaces = func(context.Context) (gonet.InterfaceStatList, error) {
+ clock = clock.Add(time.Millisecond)
+ return gonet.InterfaceStatList{{Name: "eth0", Addrs: gonet.InterfaceAddrList{{Addr: "192.0.2.10/24"}}}}, nil
+ }
+ netIOCounters = func(context.Context, bool) ([]gonet.IOCountersStat, error) {
+ netReadAt = clock
+ return []gonet.IOCountersStat{{Name: "eth0", BytesRecv: 3 << 20, BytesSent: 4 << 20}}, nil
+ }
+ ioCountersReadNow = func() time.Time { return clock }
+
+ var collector Collector
+ snapshot, err := collector.CollectWithDiskFilters(context.Background(), nil, nil)
+ if err != nil {
+ t.Fatalf("CollectWithDiskFilters: %v", err)
+ }
+ if len(snapshot.DiskIO) != 1 || len(snapshot.Network) != 1 {
+ t.Fatalf("snapshot counters = disk %+v network %+v, want one of each", snapshot.DiskIO, snapshot.Network)
+ }
+ if !snapshot.IOCountersCollectedAt.Equal(netReadAt) {
+ t.Fatalf("IOCountersCollectedAt = %v, want the network counter read at %v", snapshot.IOCountersCollectedAt, netReadAt)
+ }
+ if lag := snapshot.IOCountersCollectedAt.Sub(diskReadAt); lag < 0 || lag > 2*time.Millisecond {
+ t.Fatalf("stamp is %v after the disk counter read, want only the interface listing between them", lag)
+ }
+ if snapshot.IOCountersCollectedAt.Sub(start) < 20*time.Second {
+ t.Fatalf("stamp %v predates the stalled memory read; it must mark the counter reads", snapshot.IOCountersCollectedAt)
+ }
+}
diff --git a/internal/monitoring/agent_report_counter_clock_test.go b/internal/monitoring/agent_report_counter_clock_test.go
new file mode 100644
index 0000000000..f8609cf74c
--- /dev/null
+++ b/internal/monitoring/agent_report_counter_clock_test.go
@@ -0,0 +1,502 @@
+package monitoring
+
+import (
+ "fmt"
+ "math"
+ "reflect"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"
+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
+)
+
+// counterClockAgent is a fake Unified Agent. It collects once per interval on
+// its own clock and its cumulative counters grow at constant rates, so every
+// pair of its samples has one true rate. Pulse receipt time is whatever the
+// test's back-to-back calls take: the reconnect burst of a drained FIFO.
+type counterClockAgent struct {
+ start time.Time
+ interval time.Duration
+}
+
+const (
+ counterClockNetInPerSecond = 1_000_000
+ counterClockNetOutPerSecond = 250_000
+ counterClockReadPerSecond = 4_000_000
+ counterClockWritePerSecond = 2_000_000
+ // Busy milliseconds per second: ten per cent of the interval.
+ counterClockBusyMsPerSecond = 100
+)
+
+var counterClockTrueRates = map[string]float64{
+ "agent netin": counterClockNetInPerSecond,
+ "agent netout": counterClockNetOutPerSecond,
+ "agent diskread": counterClockReadPerSecond,
+ "agent diskwrite": counterClockWritePerSecond,
+ "sda diskread": counterClockReadPerSecond,
+ "sda diskwrite": counterClockWritePerSecond,
+ "sda disk": counterClockBusyMsPerSecond / 10.0,
+}
+
+func newCounterClockAgent(start time.Time) counterClockAgent {
+ return counterClockAgent{start: start.UTC().Truncate(time.Second), interval: 30 * time.Second}
+}
+
+func (a counterClockAgent) collectedAt(slot int) time.Time {
+ return a.start.Add(time.Duration(slot) * a.interval)
+}
+
+func (a counterClockAgent) counter(perSecond uint64, slot int) uint64 {
+ return 1_000_000_000_000 + perSecond*uint64(slot)*uint64(a.interval/time.Second)
+}
+
+// hostReport is the report holding slot's counters, authored at authoredAt
+// and sequenced in stream. An empty stream is a legacy report, and a zero
+// authoredAt a report without a timestamp. Host.UptimeSeconds marks the slot.
+func (a counterClockAgent) hostReport(stream string, seq uint64, slot int, authoredAt time.Time) agentshost.Report {
+ return agentshost.Report{
+ Agent: agentshost.AgentInfo{ID: "counter-clock-agent", Version: "6.6.0", Type: "unified", IntervalSeconds: int(a.interval / time.Second)},
+ Host: agentshost.HostInfo{
+ ID: "counter-clock-host",
+ MachineID: "counter-clock-machine",
+ Hostname: "counter-clock-host",
+ Platform: "linux",
+ UptimeSeconds: int64(slot),
+ },
+ Network: []agentshost.NetworkInterface{{
+ Name: "eth0",
+ RXBytes: a.counter(counterClockNetInPerSecond, slot),
+ TXBytes: a.counter(counterClockNetOutPerSecond, slot),
+ }},
+ DiskIO: []agentshost.DiskIO{{
+ Device: "sda",
+ ReadBytes: a.counter(counterClockReadPerSecond, slot),
+ WriteBytes: a.counter(counterClockWritePerSecond, slot),
+ IOTime: a.counter(counterClockBusyMsPerSecond, slot),
+ }},
+ Timestamp: authoredAt,
+ SequenceID: agentshost.FormatReportSequenceID(stream, seq),
+ }
+}
+
+func newCounterClockHostMonitor(t *testing.T) *Monitor {
+ t.Helper()
+ previous := mock.IsMockEnabled()
+ mustSetMockEnabled(t, false)
+ t.Cleanup(func() { mustSetMockEnabled(t, previous) })
+ return &Monitor{
+ state: models.NewState(),
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil)),
+ hostContinuityStore: config.NewHostContinuityStore(t.TempDir(), nil),
+ rateTracker: NewRateTracker(),
+ metricsHistory: NewMetricsHistory(1000, 24*time.Hour),
+ }
+}
+
+// applyCounterClockHost applies report and requires it to own host state.
+func applyCounterClockHost(t *testing.T, m *Monitor, report agentshost.Report) models.Host {
+ t.Helper()
+ host, err := m.ApplyHostReport(report, nil)
+ if err != nil {
+ t.Fatalf("ApplyHostReport slot %d: %v", report.Host.UptimeSeconds, err)
+ }
+ if host.UptimeSeconds != report.Host.UptimeSeconds {
+ t.Fatalf("slot %d report %q was not applied; host holds slot %d", report.Host.UptimeSeconds, report.SequenceID, host.UptimeSeconds)
+ }
+ return host
+}
+
+func wantTrueRate(t *testing.T, what string, got, want float64) {
+ t.Helper()
+ if math.Abs(got-want) > want*1e-9 {
+ t.Fatalf("%s = %.1f, want the agent's real rate %.1f", what, got, want)
+ }
+}
+
+func wantHostTrueRates(t *testing.T, label string, host models.Host) {
+ t.Helper()
+ wantTrueRate(t, label+" NetInRate", host.NetInRate, counterClockNetInPerSecond)
+ wantTrueRate(t, label+" NetOutRate", host.NetOutRate, counterClockNetOutPerSecond)
+ wantTrueRate(t, label+" DiskReadRate", host.DiskReadRate, counterClockReadPerSecond)
+ wantTrueRate(t, label+" DiskWriteRate", host.DiskWriteRate, counterClockWritePerSecond)
+}
+
+// wantHostRatesUnknown requires host to carry no rate. Every fake counter
+// moves, so a measured rate would be positive.
+func wantHostRatesUnknown(t *testing.T, label string, host models.Host) {
+ t.Helper()
+ if host.NetInRate != 0 || host.NetOutRate != 0 || host.DiskReadRate != 0 || host.DiskWriteRate != 0 {
+ t.Fatalf("%s reported rates without a comparable interval: in %.1f out %.1f read %.1f write %.1f",
+ label, host.NetInRate, host.NetOutRate, host.DiskReadRate, host.DiskWriteRate)
+ }
+}
+
+// counterClockHistory returns every rate point recorded for the host and its
+// sda disk, by series.
+func counterClockHistory(m *Monitor, hostID string) map[string][]float64 {
+ series := make(map[string][]float64)
+ for _, metric := range []string{"netin", "netout", "diskread", "diskwrite"} {
+ for _, point := range m.metricsHistory.GetGuestMetrics("agent:"+hostID, metric, 24*time.Hour) {
+ series["agent "+metric] = append(series["agent "+metric], point.Value)
+ }
+ }
+ diskID := hostID + ":" + normalizeHostDiskDevice("sda")
+ for _, metric := range []string{"diskread", "diskwrite", "disk"} {
+ for _, point := range m.metricsHistory.GetDiskMetrics(diskID, metric, 24*time.Hour) {
+ series["sda "+metric] = append(series["sda "+metric], point.Value)
+ }
+ }
+ return series
+}
+
+// wantHistoryTrueRates requires every rate series to hold points and every
+// point to be the agent's real rate.
+func wantHistoryTrueRates(t *testing.T, m *Monitor, hostID string) {
+ t.Helper()
+ series := counterClockHistory(m, hostID)
+ for name, want := range counterClockTrueRates {
+ if len(series[name]) == 0 {
+ t.Fatalf("%s history is empty", name)
+ }
+ for i, got := range series[name] {
+ wantTrueRate(t, fmt.Sprintf("%s history point %d", name, i), got, want)
+ }
+ }
+}
+
+// A FIFO drained on reconnect delivers reports collected 30s apart within
+// milliseconds of each other. Each is newer than the last accepted report, so
+// the stream watermark admits them all, and each must measure its counters
+// over the 30s it really covers rather than over the receipt gap.
+func TestApplyHostReportDrainedFIFORatesUseAuthoredCollectionInterval(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ // Collected two days ago and held through the outage, so rate-series
+ // cleanup must judge freshness by receipt, not by the sample clock.
+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))
+
+ first := applyCounterClockHost(t, m, agent.hostReport("stream-a", 1, 1, agent.collectedAt(1)))
+ wantHostRatesUnknown(t, "first sample", first)
+ for slot := 2; slot <= 11; slot++ {
+ host := applyCounterClockHost(t, m, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)))
+ wantHostTrueRates(t, fmt.Sprintf("drained slot %d", slot), host)
+ }
+ wantHistoryTrueRates(t, m, first.ID)
+
+ // Every series was received just now. Expiring it by its two-day-old
+ // sample time would leave the next report without a baseline.
+ m.cleanupRateTracker(time.Now())
+ next := applyCounterClockHost(t, m, agent.hostReport("stream-a", 12, 12, agent.collectedAt(12)))
+ wantHostTrueRates(t, "report after rate-series cleanup", next)
+}
+
+// An agent restart opens a new stream on the same machine clock. Its first
+// report is measured from the old stream's last accepted report, and a
+// delayed old-stream report is rejected without touching rates or history.
+func TestApplyHostReportStreamChangeKeepsMachineClockAndRetiresOldStream(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ for slot := 1; slot <= 3; slot++ {
+ applyCounterClockHost(t, m, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)))
+ }
+
+ restarted := applyCounterClockHost(t, m, agent.hostReport("stream-b", 1, 4, agent.collectedAt(4)))
+ wantHostTrueRates(t, "first report of the new stream", restarted)
+
+ before := counterClockHistory(m, restarted.ID)
+ delayed, err := m.ApplyHostReport(agent.hostReport("stream-a", 4, 3, agent.collectedAt(3)), nil)
+ if err != nil {
+ t.Fatalf("ApplyHostReport delayed old-stream report: %v", err)
+ }
+ if delayed.UptimeSeconds != 4 {
+ t.Fatalf("retired-stream report replaced state: host holds slot %d, want 4", delayed.UptimeSeconds)
+ }
+ wantHostTrueRates(t, "stored host returned for the retired stream", delayed)
+ if after := counterClockHistory(m, restarted.ID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("retired-stream report wrote rate history:\nbefore %v\nafter %v", before, after)
+ }
+
+ next := applyCounterClockHost(t, m, agent.hostReport("stream-b", 2, 5, agent.collectedAt(5)))
+ wantHostTrueRates(t, "second report of the new stream", next)
+ wantHistoryTrueRates(t, m, next.ID)
+}
+
+// Each clock transition either has a comparable interval on one clock or has
+// none. Without one, no rate is reported and no rate history is written:
+// neither an authored time subtracted from a receipt baseline, nor a huge or
+// zero rate from an interval that went backwards or did not advance.
+func TestApplyHostReportClockTransitionsReportUnknownWithoutComparableInterval(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))
+ steppedBack := func(slot int) time.Time { return agent.collectedAt(slot).Add(-time.Hour) }
+ steppedForward := func(slot int) time.Time { return agent.collectedAt(slot).Add(time.Hour) }
+
+ // A legacy report sets a receipt-time baseline.
+ hostID := applyCounterClockHost(t, m, agent.hostReport("", 0, 1, agent.collectedAt(1))).ID
+
+ steps := []struct {
+ name string
+ seq uint64
+ slot int
+ authoredAt time.Time
+ want string
+ }{
+ {"upgrade to a sequenced stream after a receipt-time baseline", 1, 2, agent.collectedAt(2), "unknown"},
+ {"second authored sample", 2, 3, agent.collectedAt(3), "true"},
+ {"sequenced report without a timestamp", 3, 4, time.Time{}, "unknown"},
+ {"timestamp returns", 4, 5, agent.collectedAt(5), "unknown"},
+ {"authored interval again", 5, 6, agent.collectedAt(6), "true"},
+ {"clock steps back an hour", 6, 7, steppedBack(7), "unknown"},
+ {"first sample on the stepped clock", 7, 8, steppedBack(8), "unknown"},
+ {"interval on the stepped clock", 8, 9, steppedBack(9), "true"},
+ {"equal timestamp with the next sequence", 9, 10, steppedBack(9), "unknown"},
+ {"first sample after the repeated timestamp", 10, 11, steppedBack(11), "unknown"},
+ {"interval after the repeated timestamp", 11, 12, steppedBack(12), "true"},
+ // A forward step is indistinguishable from a longer collection gap,
+ // which rates average over: the sample may under-report, never spike.
+ {"clock steps forward two hours", 12, 13, steppedForward(13), "no spike"},
+ {"interval on the forward clock", 13, 14, steppedForward(14), "true"},
+ }
+ for _, step := range steps {
+ before := counterClockHistory(m, hostID)
+ host := applyCounterClockHost(t, m, agent.hostReport("stream-a", step.seq, step.slot, step.authoredAt))
+ switch step.want {
+ case "unknown":
+ wantHostRatesUnknown(t, step.name, host)
+ if after := counterClockHistory(m, hostID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("%s wrote rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)
+ }
+ case "true":
+ wantHostTrueRates(t, step.name, host)
+ case "no spike":
+ for _, rate := range []struct {
+ what string
+ got, want float64
+ }{
+ {"NetInRate", host.NetInRate, counterClockNetInPerSecond},
+ {"NetOutRate", host.NetOutRate, counterClockNetOutPerSecond},
+ {"DiskReadRate", host.DiskReadRate, counterClockReadPerSecond},
+ {"DiskWriteRate", host.DiskWriteRate, counterClockWritePerSecond},
+ } {
+ if rate.got <= 0 || rate.got > rate.want {
+ t.Fatalf("%s %s = %.1f, want a positive rate no higher than the real %.1f", step.name, rate.what, rate.got, rate.want)
+ }
+ }
+ }
+ }
+ for name, values := range counterClockHistory(m, hostID) {
+ for i, got := range values {
+ if got <= 0 || got > counterClockTrueRates[name]*(1+1e-9) {
+ t.Fatalf("%s history point %d = %.1f, want a positive rate no higher than %.1f", name, i, got, counterClockTrueRates[name])
+ }
+ }
+ }
+}
+
+// Reports delivered at the same time race for the per-host lock. Whatever
+// order they win in, the watermark admits an increasing subset, each accepted
+// report is measured over the authored interval since the previously accepted
+// one, a rejected report returns the stored host, and the newest report owns
+// state.
+func TestApplyHostReportConcurrentDeliveryMeasuresAcceptedReportsOnAuthoredClock(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ first := applyCounterClockHost(t, m, agent.hostReport("stream-a", 1, 1, agent.collectedAt(1)))
+
+ const last = 16
+ results := make([]models.Host, last+1)
+ errs := make([]error, last+1)
+ release := make(chan struct{})
+ var wg sync.WaitGroup
+ for slot := 2; slot <= last; slot++ {
+ wg.Add(1)
+ go func(slot int) {
+ defer wg.Done()
+ <-release
+ results[slot], errs[slot] = m.ApplyHostReport(agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)), nil)
+ }(slot)
+ }
+ close(release)
+ wg.Wait()
+
+ for slot := 2; slot <= last; slot++ {
+ if errs[slot] != nil {
+ t.Fatalf("ApplyHostReport slot %d: %v", slot, errs[slot])
+ }
+ wantHostTrueRates(t, fmt.Sprintf("concurrent slot %d", slot), results[slot])
+ }
+ stored, ok := m.state.GetHost(first.ID)
+ if !ok || stored.UptimeSeconds != last {
+ t.Fatalf("newest report must own state: found %v holding slot %d, want slot %d", ok, stored.UptimeSeconds, last)
+ }
+ wantHostTrueRates(t, "stored host", stored)
+ wantHistoryTrueRates(t, m, first.ID)
+}
+
+// Legacy reports carry no stream, so their timestamps only guard the
+// reconnect burst and never time a rate: rates stay on Pulse receipt time.
+func TestApplyHostReportLegacyRatesStayOnReceiptTime(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ host := applyCounterClockHost(t, m, agent.hostReport("", 0, 1, agent.collectedAt(1)))
+
+ // Unchanged counters are measured idle over the receipt interval.
+ idle := agent.hostReport("", 0, 1, agent.collectedAt(2))
+ idle.Host.UptimeSeconds = 2
+ idleSentAt := time.Now()
+ applyCounterClockHost(t, m, idle)
+ points := m.metricsHistory.GetGuestMetrics("agent:"+host.ID, "netin", 24*time.Hour)
+ if len(points) != 1 || points[0].Value != 0 {
+ t.Fatalf("unchanged legacy counters must be measured idle: %+v", points)
+ }
+
+ // Two slots of counters authored one interval later: an authored-clock
+ // rate would be twice the real rate, while the receipt interval is at
+ // most the wall time these calls took.
+ moved := applyCounterClockHost(t, m, agent.hostReport("", 0, 3, agent.collectedAt(3)))
+ wall := time.Since(idleSentAt).Seconds()
+ delta := float64(agent.counter(counterClockNetInPerSecond, 3) - agent.counter(counterClockNetInPerSecond, 1))
+ if moved.NetInRate < delta/wall {
+ t.Fatalf("legacy NetInRate = %.1f, want the receipt-interval rate of at least %.1f (authored interval would give %.1f)",
+ moved.NetInRate, delta/wall, delta/agent.interval.Seconds())
+ }
+
+ // A backwards legacy timestamp inside the reconnect burst stays rejected.
+ before := counterClockHistory(m, host.ID)
+ late := agent.hostReport("", 0, 2, agent.collectedAt(2))
+ stored, err := m.ApplyHostReport(late, nil)
+ if err != nil {
+ t.Fatalf("ApplyHostReport late legacy report: %v", err)
+ }
+ if stored.UptimeSeconds != 3 {
+ t.Fatalf("late legacy report replaced state: host holds slot %d, want 3", stored.UptimeSeconds)
+ }
+ if after := counterClockHistory(m, host.ID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("late legacy report wrote rate history:\nbefore %v\nafter %v", before, after)
+ }
+}
+
+// The Docker module follows the same stream ordering. A drained FIFO measures
+// container counters on the authored interval, and a helper-degraded status
+// report between complete reports neither replaces the inventory nor moves a
+// rate baseline, so the next complete report spans both intervals.
+func TestApplyDockerReportDrainedFIFOContainerRatesUseAuthoredCollectionInterval(t *testing.T) {
+ previous := mock.IsMockEnabled()
+ mustSetMockEnabled(t, false)
+ t.Cleanup(func() { mustSetMockEnabled(t, previous) })
+ m := newTestMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+
+ report := func(seq uint64, slot int, complete bool) agentsdocker.Report {
+ inventoryComplete := complete
+ r := agentsdocker.Report{
+ Agent: agentsdocker.AgentInfo{ID: "counter-clock-docker", Version: "6.6.0", Type: "unified", IntervalSeconds: 30},
+ Host: agentsdocker.HostInfo{Hostname: "counter-clock-docker-host"},
+ InventoryComplete: &inventoryComplete,
+ Timestamp: agent.collectedAt(slot),
+ SequenceID: agentshost.FormatReportSequenceID("docker-stream", seq),
+ }
+ if complete {
+ r.Containers = []agentsdocker.Container{{
+ ID: "counter-clock-container",
+ Name: "api",
+ State: "running",
+ NetworkRXBytes: agent.counter(counterClockNetInPerSecond, slot),
+ NetworkTXBytes: agent.counter(counterClockNetOutPerSecond, slot),
+ BlockIO: &agentsdocker.ContainerBlockIO{
+ ReadBytes: agent.counter(counterClockReadPerSecond, slot),
+ WriteBytes: agent.counter(counterClockWritePerSecond, slot),
+ },
+ }}
+ }
+ return r
+ }
+ apply := func(seq uint64, slot int, complete bool) models.DockerContainer {
+ t.Helper()
+ host, err := m.ApplyDockerReport(report(seq, slot, complete), nil)
+ if err != nil {
+ t.Fatalf("ApplyDockerReport slot %d: %v", slot, err)
+ }
+ if len(host.Containers) != 1 {
+ t.Fatalf("slot %d containers = %+v, want the one running container", slot, host.Containers)
+ }
+ return host.Containers[0]
+ }
+ wantContainerTrueRates := func(label string, container models.DockerContainer) {
+ t.Helper()
+ wantTrueRate(t, label+" NetInRate", container.NetInRate, counterClockNetInPerSecond)
+ wantTrueRate(t, label+" NetOutRate", container.NetOutRate, counterClockNetOutPerSecond)
+ if container.BlockIO == nil || container.BlockIO.ReadRateBytesPerSecond == nil || container.BlockIO.WriteRateBytesPerSecond == nil {
+ t.Fatalf("%s block I/O rates missing: %+v", label, container.BlockIO)
+ }
+ wantTrueRate(t, label+" read rate", *container.BlockIO.ReadRateBytesPerSecond, counterClockReadPerSecond)
+ wantTrueRate(t, label+" write rate", *container.BlockIO.WriteRateBytesPerSecond, counterClockWritePerSecond)
+ }
+ containerHistory := func() map[string][]float64 {
+ series := make(map[string][]float64)
+ for _, metric := range []string{"netin", "netout", "diskread", "diskwrite"} {
+ for _, point := range m.metricsHistory.GetGuestMetrics("docker:counter-clock-container", metric, 24*time.Hour) {
+ series[metric] = append(series[metric], point.Value)
+ }
+ }
+ return series
+ }
+
+ apply(1, 1, true)
+ for metric, values := range containerHistory() {
+ if len(values) != 0 {
+ t.Fatalf("first sample wrote %s history without an interval: %v", metric, values)
+ }
+ }
+ for slot := 2; slot <= 6; slot++ {
+ wantContainerTrueRates(fmt.Sprintf("drained slot %d", slot), apply(uint64(slot), slot, true))
+ }
+
+ before := containerHistory()
+ retained := apply(7, 7, false)
+ wantContainerTrueRates("container retained through the status-only report", retained)
+ if after := containerHistory(); !reflect.DeepEqual(after, before) {
+ t.Fatalf("status-only report wrote container history:\nbefore %v\nafter %v", before, after)
+ }
+
+ wantContainerTrueRates("complete report after the status-only report", apply(8, 8, true))
+ before = containerHistory()
+ fault := report(9, 9, true)
+ fault.Timestamp = agent.collectedAt(8)
+ if _, err := m.ApplyDockerReport(fault, nil); err != nil {
+ t.Fatal(err)
+ }
+ if after := containerHistory(); !reflect.DeepEqual(after, before) {
+ t.Fatalf("repeated source time wrote unknown rates as history: before %v after %v", before, after)
+ }
+ apply(10, 10, true) // First baseline after dropping the ambiguous sample.
+ wantContainerTrueRates("interval after the clock fault", apply(11, 11, true))
+ series := containerHistory()
+ for metric, want := range map[string]float64{
+ "netin": counterClockNetInPerSecond, "netout": counterClockNetOutPerSecond,
+ "diskread": counterClockReadPerSecond, "diskwrite": counterClockWritePerSecond,
+ } {
+ values := series[metric]
+ if len(values) == 0 {
+ t.Fatalf("container %s history is empty", metric)
+ }
+ for i, got := range values {
+ wantTrueRate(t, fmt.Sprintf("container %s history point %d", metric, i), got, want)
+ }
+ }
+ idle := report(12, 12, true)
+ idle.Containers = report(11, 11, true).Containers
+ if _, err := m.ApplyDockerReport(idle, nil); err != nil {
+ t.Fatal(err)
+ }
+ for metric, values := range containerHistory() {
+ if len(values) == 0 || values[len(values)-1] != 0 {
+ t.Fatalf("measured idle %s did not retain zero: %v", metric, values)
+ }
+ }
+}
diff --git a/internal/monitoring/agent_report_io_counter_clock_test.go b/internal/monitoring/agent_report_io_counter_clock_test.go
new file mode 100644
index 0000000000..004fb0b906
--- /dev/null
+++ b/internal/monitoring/agent_report_io_counter_clock_test.go
@@ -0,0 +1,170 @@
+package monitoring
+
+import (
+ "encoding/json"
+ "fmt"
+ "reflect"
+ "testing"
+ "time"
+
+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
+)
+
+// ioCounterStamp is the JSON value of a counter read stamp.
+func ioCounterStamp(t *testing.T, at time.Time) string {
+ t.Helper()
+ raw, err := json.Marshal(at.UTC())
+ if err != nil {
+ t.Fatalf("encode counter read stamp: %v", err)
+ }
+ return string(raw)
+}
+
+// ioCounterWireReport decodes report from the JSON a current agent sends,
+// with stamp spliced in as its ioCountersCollectedAt value; an empty stamp is
+// an agent that sends none. A server without the field ignores it, so these
+// tests compile there and fail on its behaviour.
+func ioCounterWireReport(t *testing.T, report agentshost.Report, stamp string) agentshost.Report {
+ t.Helper()
+ raw, err := json.Marshal(report)
+ if err != nil {
+ t.Fatalf("encode report: %v", err)
+ }
+ if stamp != "" {
+ raw = append(raw[:len(raw)-1], []byte(`,"ioCountersCollectedAt":`+stamp+`}`)...)
+ }
+ var decoded agentshost.Report
+ if err := json.Unmarshal(raw, &decoded); err != nil {
+ t.Fatalf("decode wire report: %v", err)
+ }
+ return decoded
+}
+
+// slowTail is how long optional collectors ran after the counter read before
+// the report Timestamp: 20s on even slots, none on odd ones.
+func slowTail(slot int) time.Duration {
+ if slot%2 == 0 {
+ return 20 * time.Second
+ }
+ return 0
+}
+
+// Counters are read every 30s at 0/30/60/...; optional collectors then delay
+// the report Timestamp to 0/50/60/110/... A drained FIFO delivers the reports
+// back to back. Rates must divide by the 30s between counter reads, not by
+// the 50s/10s between Timestamps or by the receipt gap.
+func TestApplyHostReportSlowOptionalTailRatesUseIOCounterReadTime(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))
+ report := func(slot int) agentshost.Report {
+ authored := agent.collectedAt(slot).Add(slowTail(slot))
+ return ioCounterWireReport(t, agent.hostReport("stream-a", uint64(slot), slot, authored), ioCounterStamp(t, agent.collectedAt(slot)))
+ }
+
+ first := applyCounterClockHost(t, m, report(1))
+ wantHostRatesUnknown(t, "first sample", first)
+ for slot := 2; slot <= 11; slot++ {
+ host := applyCounterClockHost(t, m, report(slot))
+ wantHostTrueRates(t, fmt.Sprintf("drained slot %d", slot), host)
+ }
+ wantHistoryTrueRates(t, m, first.ID)
+
+ // Received just now from two-day-old reads: cleanup keyed on the sample
+ // clock would drop the baselines.
+ m.cleanupRateTracker(time.Now())
+ wantHostTrueRates(t, "report after rate-series cleanup", applyCounterClockHost(t, m, report(12)))
+}
+
+// Negative control: the same slow-tail reports from an agent that sends no
+// counter read stamp keep the report-Timestamp fallback, so their rates carry
+// the tail (30/50 and 30/10 of the real rate). This proves the tail above
+// reaches the denominator, and that an older agent is left as it was.
+func TestApplyHostReportOlderAgentSlowTailKeepsTimestampFallback(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ report := func(slot int) agentshost.Report {
+ return ioCounterWireReport(t, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot).Add(slowTail(slot))), "")
+ }
+ applyCounterClockHost(t, m, report(1))
+ slow := applyCounterClockHost(t, m, report(2))
+ wantTrueRate(t, "older agent after a slow tail", slow.NetInRate, counterClockNetInPerSecond*30.0/50)
+ fast := applyCounterClockHost(t, m, report(3))
+ wantTrueRate(t, "older agent after the tail ends", fast.NetInRate, counterClockNetInPerSecond*30.0/10)
+}
+
+// Every sampling-mode transition and source clock fault either has a
+// comparable interval in one mode or reports no rate and writes no rate
+// history: report-Timestamp and counter-read times share a clock but not an
+// offset, so a switch is a new baseline, and a stamp that does not advance
+// drops it. A counter reset keeps its existing rebase to zero.
+func TestApplyHostReportIOCounterReadTimeTransitions(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))
+ at := agent.collectedAt
+ back := func(slot int) time.Time { return at(slot).Add(-time.Hour) }
+ stamp := func(readAt time.Time) string { return ioCounterStamp(t, readAt) }
+
+ // A legacy report sets receipt-time baselines; its stamp has no ordered
+ // clock behind it.
+ legacy := ioCounterWireReport(t, agent.hostReport("", 0, 1, at(1)), stamp(at(1)))
+ hostID := applyCounterClockHost(t, m, legacy).ID
+
+ steps := []struct {
+ name string
+ stream string
+ seq uint64
+ slot int
+ stamp string
+ want string
+ }{
+ {"older sequenced agent after receipt-time baselines", "stream-a", 1, 2, "", "unknown"},
+ {"older agent on its report Timestamp", "stream-a", 2, 3, "", "true"},
+ {"agent starts stamping its counter read", "stream-a", 3, 4, stamp(at(4)), "unknown"},
+ {"counter-read interval", "stream-a", 4, 5, stamp(at(5)), "true"},
+ {"stamp missing", "stream-a", 5, 6, "", "unknown"},
+ {"stamp returns", "stream-a", 6, 7, stamp(at(7)), "unknown"},
+ {"counter-read interval again", "stream-a", 7, 8, stamp(at(8)), "true"},
+ {"zero stamp", "stream-a", 8, 9, `"0001-01-01T00:00:00Z"`, "unknown"},
+ {"stamp after the zero one", "stream-a", 9, 10, stamp(at(10)), "unknown"},
+ {"interval after the zero stamp", "stream-a", 10, 11, stamp(at(11)), "true"},
+ {"agent restart keeps the machine clock", "stream-b", 1, 12, stamp(at(12)), "true"},
+ {"stamp steps back an hour", "stream-b", 2, 13, stamp(back(13)), "unknown"},
+ {"first read on the stepped clock", "stream-b", 3, 14, stamp(back(14)), "unknown"},
+ {"interval on the stepped clock", "stream-b", 4, 15, stamp(back(15)), "true"},
+ {"equal stamp with the next sequence", "stream-b", 5, 16, stamp(back(15)), "unknown"},
+ {"first read after the repeated stamp", "stream-b", 6, 17, stamp(back(17)), "unknown"},
+ {"interval after the repeated stamp", "stream-b", 7, 18, stamp(back(18)), "true"},
+ }
+ for _, step := range steps {
+ before := counterClockHistory(m, hostID)
+ // Older-agent steps carry no tail so their Timestamp interval is exact.
+ report := ioCounterWireReport(t, agent.hostReport(step.stream, step.seq, step.slot, at(step.slot)), step.stamp)
+ host := applyCounterClockHost(t, m, report)
+ switch step.want {
+ case "unknown":
+ wantHostRatesUnknown(t, step.name, host)
+ if after := counterClockHistory(m, hostID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("%s wrote rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)
+ }
+ case "true":
+ wantHostTrueRates(t, step.name, host)
+ }
+ }
+
+ // The host reboots: every counter restarts from zero. That keeps its
+ // existing meaning, a measured zero that rebases, and the next read is
+ // measured from the restarted counters.
+ rebooted := agent.hostReport("stream-b", 8, 19, back(19))
+ rebooted.Network[0].RXBytes, rebooted.Network[0].TXBytes = 0, 0
+ rebooted.DiskIO[0].ReadBytes, rebooted.DiskIO[0].WriteBytes, rebooted.DiskIO[0].IOTime = 0, 0, 0
+ host := applyCounterClockHost(t, m, ioCounterWireReport(t, rebooted, stamp(back(19))))
+ points := m.metricsHistory.GetGuestMetrics("agent:"+hostID, "netin", 24*time.Hour)
+ if host.NetInRate != 0 || len(points) == 0 || points[len(points)-1].Value != 0 {
+ t.Fatalf("counter reset = rate %.1f, last history %+v, want the existing measured-zero rebase", host.NetInRate, points)
+ }
+ next := agent.hostReport("stream-b", 9, 20, back(20))
+ slots := uint64(agent.interval / time.Second)
+ next.Network[0].RXBytes, next.Network[0].TXBytes = counterClockNetInPerSecond*slots, counterClockNetOutPerSecond*slots
+ next.DiskIO[0].ReadBytes, next.DiskIO[0].WriteBytes, next.DiskIO[0].IOTime = counterClockReadPerSecond*slots, counterClockWritePerSecond*slots, counterClockBusyMsPerSecond*slots
+ wantHostTrueRates(t, "first interval after the reset", applyCounterClockHost(t, m, ioCounterWireReport(t, next, stamp(back(20)))))
+}
diff --git a/internal/monitoring/agent_report_libvirt_clock_test.go b/internal/monitoring/agent_report_libvirt_clock_test.go
new file mode 100644
index 0000000000..5bcf3c2590
--- /dev/null
+++ b/internal/monitoring/agent_report_libvirt_clock_test.go
@@ -0,0 +1,326 @@
+package monitoring
+
+import (
+ "fmt"
+ "math"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
+)
+
+const (
+ // One vCPU-second per second across four vCPUs: 25% of capacity.
+ libvirtClockCPUNanosPerSecond = 1_000_000_000
+ libvirtClockVCPUs = 4
+ libvirtClockNetInPerSecond = 500_000
+ libvirtClockNetOutPerSecond = 125_000
+ libvirtClockReadPerSecond = 2_000_000
+ libvirtClockWritePerSecond = 1_000_000
+ // The agent stamps CollectedAt before it reads domain stats and the report
+ // Timestamp after every collector, so CollectedAt sits earlier.
+ libvirtClockCollectedLead = 7 * time.Second
+)
+
+var libvirtClockTrueRates = map[string]float64{
+ "cpu": 100.0 / libvirtClockVCPUs,
+ "netin": libvirtClockNetInPerSecond,
+ "netout": libvirtClockNetOutPerSecond,
+ "diskread": libvirtClockReadPerSecond,
+ "diskwrite": libvirtClockWritePerSecond,
+}
+
+func TestApplyHostReportLibvirtCPUCounterRetainsUnsignedRange(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ for slot := 1; slot <= 3; slot++ {
+ domain := agent.libvirtDomain(slot, 0)
+ // A busy, long-running domain can accumulate more than MaxInt64
+ // nanoseconds across its vCPUs. Clamping both samples loses the delta.
+ domain.CPUTimeNanoseconds = uint64(math.MaxInt64) - 15_000_000_000 + uint64(slot)*30_000_000_000
+ report := agent.libvirtHostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), domain)
+ got := libvirtClockDomain(t, "unsigned CPU counter", applyCounterClockHost(t, m, report))
+ if slot > 1 {
+ wantLibvirtTrueRates(t, "unsigned CPU counter", got)
+ }
+ }
+}
+
+// libvirtCollectedAt is when the fake agent read slot's domain stats.
+func (a counterClockAgent) libvirtCollectedAt(slot int) time.Time {
+ return a.collectedAt(slot).Add(-libvirtClockCollectedLead)
+}
+
+// libvirtCounter is a domain counter at slot. A domain restarted at
+// restartSlot counts again from zero; restartSlot 0 means it never restarted.
+func (a counterClockAgent) libvirtCounter(perSecond uint64, slot, restartSlot int) uint64 {
+ if restartSlot == 0 {
+ return a.counter(perSecond, slot)
+ }
+ return perSecond * uint64(slot-restartSlot) * uint64(a.interval/time.Second)
+}
+
+func (a counterClockAgent) libvirtDomain(slot, restartSlot int) agentshost.LibvirtDomain {
+ return agentshost.LibvirtDomain{
+ Name: "app",
+ State: "running",
+ VCPUs: libvirtClockVCPUs,
+ CPUTimeNanoseconds: a.libvirtCounter(libvirtClockCPUNanosPerSecond, slot, restartSlot),
+ MemoryCurrentBytes: 2 << 30,
+ MemoryMaximumBytes: 4 << 30,
+ NetworkRXBytes: a.libvirtCounter(libvirtClockNetInPerSecond, slot, restartSlot),
+ NetworkTXBytes: a.libvirtCounter(libvirtClockNetOutPerSecond, slot, restartSlot),
+ DiskReadBytes: a.libvirtCounter(libvirtClockReadPerSecond, slot, restartSlot),
+ DiskWriteBytes: a.libvirtCounter(libvirtClockWritePerSecond, slot, restartSlot),
+ }
+}
+
+// libvirtHostReport is hostReport carrying a readable libvirt inventory
+// collected at collectedAt (zero: an inventory without CollectedAt), holding
+// the "app" domain unless it is omitted.
+func (a counterClockAgent) libvirtHostReport(stream string, seq uint64, slot int, authoredAt, collectedAt time.Time, domains ...agentshost.LibvirtDomain) agentshost.Report {
+ report := a.hostReport(stream, seq, slot, authoredAt)
+ report.Libvirt = &agentshost.LibvirtInventory{
+ Domains: append([]agentshost.LibvirtDomain{}, domains...),
+ CollectedAt: collectedAt,
+ }
+ return report
+}
+
+func libvirtClockDomain(t *testing.T, label string, host models.Host) models.HostLibvirtDomain {
+ t.Helper()
+ if host.Libvirt == nil || len(host.Libvirt.Domains) != 1 {
+ t.Fatalf("%s libvirt inventory = %+v, want the app domain", label, host.Libvirt)
+ }
+ return host.Libvirt.Domains[0]
+}
+
+func wantLibvirtTrueRates(t *testing.T, label string, domain models.HostLibvirtDomain) {
+ t.Helper()
+ if !domain.CPUUsageValid || !domain.IORatesValid {
+ t.Fatalf("%s reported no rate (cpu valid %v, io valid %v), want the domain's real rates", label, domain.CPUUsageValid, domain.IORatesValid)
+ }
+ wantTrueRate(t, label+" CPU usage", domain.CPUUsagePercent, libvirtClockTrueRates["cpu"])
+ wantTrueRate(t, label+" NetworkInRate", domain.NetworkInRate, libvirtClockNetInPerSecond)
+ wantTrueRate(t, label+" NetworkOutRate", domain.NetworkOutRate, libvirtClockNetOutPerSecond)
+ wantTrueRate(t, label+" DiskReadRate", domain.DiskReadRate, libvirtClockReadPerSecond)
+ wantTrueRate(t, label+" DiskWriteRate", domain.DiskWriteRate, libvirtClockWritePerSecond)
+}
+
+func wantLibvirtUnknown(t *testing.T, label string, domain models.HostLibvirtDomain) {
+ t.Helper()
+ if domain.CPUUsageValid || domain.IORatesValid {
+ t.Fatalf("%s reported rates without a comparable interval: cpu %.1f (valid %v), in %.1f (valid %v)",
+ label, domain.CPUUsagePercent, domain.CPUUsageValid, domain.NetworkInRate, domain.IORatesValid)
+ }
+}
+
+// libvirtClockHistory returns every CPU and I/O history point recorded for
+// the app domain, by metric.
+func libvirtClockHistory(m *Monitor, hostID string) map[string][]float64 {
+ metricID := libvirtDomainMetricID(hostID, agentLibvirtDomainID("app"))
+ series := make(map[string][]float64)
+ for metric := range libvirtClockTrueRates {
+ for _, point := range m.metricsHistory.GetGuestMetrics(metricID, metric, 24*time.Hour) {
+ series[metric] = append(series[metric], point.Value)
+ }
+ }
+ return series
+}
+
+func wantLibvirtHistoryTrueRates(t *testing.T, m *Monitor, hostID string) {
+ t.Helper()
+ series := libvirtClockHistory(m, hostID)
+ for metric, want := range libvirtClockTrueRates {
+ if len(series[metric]) == 0 {
+ t.Fatalf("libvirt %s history is empty", metric)
+ }
+ for i, got := range series[metric] {
+ wantTrueRate(t, fmt.Sprintf("libvirt %s history point %d", metric, i), got, want)
+ }
+ }
+}
+
+// A drained FIFO delivers host reports collected 30s apart within
+// milliseconds. The libvirt domain's CPU and I/O counters inside each report
+// must be measured over the 30s between their CollectedAt stamps, not over the
+// receipt gap, and not over the report Timestamp, which the agent stamps after
+// a variable-length tail of other collectors.
+func TestApplyHostReportDrainedFIFOLibvirtRatesUseCollectedAt(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))
+ authoredAt := func(slot int) time.Time {
+ return agent.collectedAt(slot).Add(time.Duration(slot%3) * 4 * time.Second)
+ }
+ report := func(slot int) agentshost.Report {
+ return agent.libvirtHostReport("stream-a", uint64(slot), slot, authoredAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))
+ }
+
+ first := applyCounterClockHost(t, m, report(1))
+ wantLibvirtUnknown(t, "first sample", libvirtClockDomain(t, "first sample", first))
+ for slot := 2; slot <= 11; slot++ {
+ host := applyCounterClockHost(t, m, report(slot))
+ wantLibvirtTrueRates(t, fmt.Sprintf("drained slot %d", slot), libvirtClockDomain(t, "drained", host))
+ }
+ wantLibvirtHistoryTrueRates(t, m, first.ID)
+
+ // Both domain series were received just now; cleanup keyed on their
+ // two-day-old sample clock would drop the baselines.
+ m.cleanupRateTracker(time.Now())
+ next := applyCounterClockHost(t, m, report(12))
+ wantLibvirtTrueRates(t, "report after rate-series cleanup", libvirtClockDomain(t, "after cleanup", next))
+}
+
+// Each libvirt clock transition either has a comparable CollectedAt interval
+// or reports no CPU or I/O rate and writes no rate history for that sample.
+func TestApplyHostReportLibvirtClockTransitionsReportUnknownWithoutComparableInterval(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))
+ steppedBack := func(slot int) time.Time { return agent.libvirtCollectedAt(slot).Add(-time.Hour) }
+
+ // A legacy report sets receipt-time baselines even though it carries
+ // CollectedAt: without a stream it has no ordered source clock.
+ legacy := agent.libvirtHostReport("", 0, 1, agent.collectedAt(1), agent.libvirtCollectedAt(1), agent.libvirtDomain(1, 0))
+ hostID := applyCounterClockHost(t, m, legacy).ID
+
+ steps := []struct {
+ name string
+ seq uint64
+ slot int
+ authoredAt time.Time
+ collectedAt time.Time
+ want string
+ }{
+ {"upgrade to a sequenced stream after receipt-time baselines", 1, 2, agent.collectedAt(2), agent.libvirtCollectedAt(2), "unknown"},
+ {"second CollectedAt sample", 2, 3, agent.collectedAt(3), agent.libvirtCollectedAt(3), "true"},
+ {"report without a Timestamp keeps the inventory's own clock", 3, 4, time.Time{}, agent.libvirtCollectedAt(4), "true"},
+ {"inventory without CollectedAt", 4, 5, agent.collectedAt(5), time.Time{}, "unknown"},
+ {"CollectedAt returns", 5, 6, agent.collectedAt(6), agent.libvirtCollectedAt(6), "unknown"},
+ {"CollectedAt interval again", 6, 7, agent.collectedAt(7), agent.libvirtCollectedAt(7), "true"},
+ {"clock steps back an hour", 7, 8, agent.collectedAt(8).Add(-time.Hour), steppedBack(8), "unknown"},
+ {"first sample on the stepped clock", 8, 9, agent.collectedAt(9).Add(-time.Hour), steppedBack(9), "unknown"},
+ {"interval on the stepped clock", 9, 10, agent.collectedAt(10).Add(-time.Hour), steppedBack(10), "true"},
+ {"equal CollectedAt with the next sequence", 10, 11, agent.collectedAt(11).Add(-time.Hour), steppedBack(10), "unknown"},
+ {"first sample after the repeated CollectedAt", 11, 12, agent.collectedAt(12).Add(-time.Hour), steppedBack(12), "unknown"},
+ {"interval after the repeated CollectedAt", 12, 13, agent.collectedAt(13).Add(-time.Hour), steppedBack(13), "true"},
+ }
+ for _, step := range steps {
+ before := libvirtClockHistory(m, hostID)
+ host := applyCounterClockHost(t, m, agent.libvirtHostReport("stream-a", step.seq, step.slot, step.authoredAt, step.collectedAt, agent.libvirtDomain(step.slot, 0)))
+ domain := libvirtClockDomain(t, step.name, host)
+ switch step.want {
+ case "unknown":
+ wantLibvirtUnknown(t, step.name, domain)
+ if after := libvirtClockHistory(m, hostID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("%s wrote libvirt rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)
+ }
+ case "true":
+ wantLibvirtTrueRates(t, step.name, domain)
+ }
+ }
+ wantLibvirtHistoryTrueRates(t, m, hostID)
+}
+
+// An agent restart keeps the machine clock, so the new stream's first libvirt
+// sample is measured from the old stream's last, and a delayed old-stream
+// report changes neither the domain nor its history.
+func TestApplyHostReportLibvirtStreamChangeKeepsMachineClock(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ report := func(stream string, seq uint64, slot int) agentshost.Report {
+ return agent.libvirtHostReport(stream, seq, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))
+ }
+ for slot := 1; slot <= 3; slot++ {
+ applyCounterClockHost(t, m, report("stream-a", uint64(slot), slot))
+ }
+
+ restarted := applyCounterClockHost(t, m, report("stream-b", 1, 4))
+ wantLibvirtTrueRates(t, "first report of the new stream", libvirtClockDomain(t, "restarted", restarted))
+
+ before := libvirtClockHistory(m, restarted.ID)
+ delayed, err := m.ApplyHostReport(report("stream-a", 4, 3), nil)
+ if err != nil {
+ t.Fatalf("ApplyHostReport delayed old-stream report: %v", err)
+ }
+ if delayed.UptimeSeconds != 4 {
+ t.Fatalf("retired-stream report replaced state: host holds slot %d, want 4", delayed.UptimeSeconds)
+ }
+ wantLibvirtTrueRates(t, "stored domain returned for the retired stream", libvirtClockDomain(t, "delayed", delayed))
+ if after := libvirtClockHistory(m, restarted.ID); !reflect.DeepEqual(after, before) {
+ t.Fatalf("retired-stream report wrote libvirt history:\nbefore %v\nafter %v", before, after)
+ }
+
+ next := applyCounterClockHost(t, m, report("stream-b", 2, 5))
+ wantLibvirtTrueRates(t, "second report of the new stream", libvirtClockDomain(t, "next", next))
+}
+
+// Omissions keep their existing meaning: a readable inventory without the
+// domain retires it, so CPU usage restarts on its return, while its
+// cumulative I/O counters measure the whole span since their last sample; an
+// inventory the agent failed to read preserves the last one, and the next
+// read measures CPU from that last read rather than from the host report in
+// between; a domain whose counters restart has no CPU usage for that sample.
+func TestApplyHostReportLibvirtOmittedDomainInventoryAndRestart(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ report := func(seq uint64, slot int, domains ...agentshost.LibvirtDomain) agentshost.Report {
+ return agent.libvirtHostReport("stream-a", seq, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), domains...)
+ }
+ for slot := 1; slot <= 2; slot++ {
+ applyCounterClockHost(t, m, report(uint64(slot), slot, agent.libvirtDomain(slot, 0)))
+ }
+
+ omitted := applyCounterClockHost(t, m, report(3, 3))
+ if omitted.Libvirt == nil || len(omitted.Libvirt.Domains) != 0 {
+ t.Fatalf("readable inventory without the domain = %+v, want no domains", omitted.Libvirt)
+ }
+
+ returned := libvirtClockDomain(t, "returned", applyCounterClockHost(t, m, report(4, 4, agent.libvirtDomain(4, 0))))
+ if returned.CPUUsageValid {
+ t.Fatalf("domain absent from the previous report reported CPU usage %.1f", returned.CPUUsagePercent)
+ }
+ if !returned.IORatesValid {
+ t.Fatal("returned domain lost its I/O rates")
+ }
+ wantTrueRate(t, "returned NetworkInRate", returned.NetworkInRate, libvirtClockNetInPerSecond)
+ wantTrueRate(t, "returned DiskWriteRate", returned.DiskWriteRate, libvirtClockWritePerSecond)
+
+ wantLibvirtTrueRates(t, "domain after its return", libvirtClockDomain(t, "slot 5", applyCounterClockHost(t, m, report(5, 5, agent.libvirtDomain(5, 0)))))
+
+ unread := agent.hostReport("stream-a", 6, 6, agent.collectedAt(6))
+ wantLibvirtTrueRates(t, "inventory preserved through a failed read", libvirtClockDomain(t, "slot 6", applyCounterClockHost(t, m, unread)))
+ wantLibvirtTrueRates(t, "read after the failed one", libvirtClockDomain(t, "slot 7", applyCounterClockHost(t, m, report(7, 7, agent.libvirtDomain(7, 0)))))
+
+ restarted := libvirtClockDomain(t, "restarted", applyCounterClockHost(t, m, report(8, 8, agent.libvirtDomain(8, 8))))
+ if restarted.CPUUsageValid {
+ t.Fatalf("restarted domain reported CPU usage %.1f across its counter reset", restarted.CPUUsagePercent)
+ }
+ if !restarted.IORatesValid || restarted.NetworkInRate != 0 || restarted.DiskReadRate != 0 {
+ t.Fatalf("restarted domain I/O = %+v, want the counter-reset rebase of zero", restarted)
+ }
+ wantLibvirtTrueRates(t, "first interval after the restart", libvirtClockDomain(t, "slot 9", applyCounterClockHost(t, m, report(9, 9, agent.libvirtDomain(9, 8)))))
+}
+
+// Legacy reports have no ordered source clock, so their libvirt counters keep
+// receipt time even when the inventory carries CollectedAt.
+func TestApplyHostReportLegacyLibvirtRatesStayOnReceiptTime(t *testing.T) {
+ m := newCounterClockHostMonitor(t)
+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))
+ report := func(slot int) agentshost.Report {
+ return agent.libvirtHostReport("", 0, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))
+ }
+ applyCounterClockHost(t, m, report(1))
+ sentAt := time.Now()
+ domain := libvirtClockDomain(t, "legacy", applyCounterClockHost(t, m, report(2)))
+ wall := time.Since(sentAt).Seconds()
+ if !domain.CPUUsageValid || !domain.IORatesValid {
+ t.Fatalf("legacy domain lost its receipt-time rates: %+v", domain)
+ }
+ delta := float64(agent.libvirtCounter(libvirtClockNetInPerSecond, 2, 0) - agent.libvirtCounter(libvirtClockNetInPerSecond, 1, 0))
+ if domain.NetworkInRate < delta/wall {
+ t.Fatalf("legacy NetworkInRate = %.1f, want the receipt-interval rate of at least %.1f (CollectedAt would give %.1f)",
+ domain.NetworkInRate, delta/wall, delta/agent.interval.Seconds())
+ }
+}
diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go
index 31b0962645..ee784c4222 100644
--- a/internal/monitoring/canonical_guardrails_test.go
+++ b/internal/monitoring/canonical_guardrails_test.go
@@ -2133,8 +2133,9 @@ func TestHostPhysicalDiskIOMetricsUseCanonicalDiskHistoryPath(t *testing.T) {
}
source := string(data)
requiredSnippets := []string{
- "m.writeHostPhysicalDiskIOMetrics(host, now)",
- "func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time) {",
+ "m.writeHostPhysicalDiskIOMetrics(host, now, counterSampled)",
+ "func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time, sampled counterSampleTime) {",
+ "readRate, writeRate, busyPct, _, _ := m.rateTracker.calculateSampledRates(trackerKey, current, sampled)",
`resourceID := unifiedresources.HostSMARTDiskMetricID(host, disk)`,
`m.metricsHistory.AddDiskMetric(resourceID, "diskread", readRate, now)`,
`m.metricsStore.Write("disk", resourceID, "diskwrite", writeRate, now)`,
diff --git a/internal/monitoring/issue1595_collection_trust_test.go b/internal/monitoring/issue1595_collection_trust_test.go
index 79f5366ca5..1ab373fc16 100644
--- a/internal/monitoring/issue1595_collection_trust_test.go
+++ b/internal/monitoring/issue1595_collection_trust_test.go
@@ -434,7 +434,7 @@ func TestIdentitylessControllerMembersReadTheirWritersHistory(t *testing.T) {
hosts := []models.Host{standalone, linked}
for _, host := range hosts {
- m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second))
+ m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second), counterSampleTime{})
}
for _, host := range hosts {
for i := range host.DiskIO {
@@ -442,7 +442,7 @@ func TestIdentitylessControllerMembersReadTheirWritersHistory(t *testing.T) {
host.DiskIO[i].WriteBytes += 1 << 20
host.DiskIO[i].IOTime += 1000
}
- m.writeHostPhysicalDiskIOMetrics(host, now)
+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampleTime{})
m.writeHostSMARTMetrics(host, now)
}
m.writeSMARTMetrics(pveMember, now)
@@ -638,13 +638,13 @@ func TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports(t *testing.T) {
for _, device := range []string{"sdb", "sdc", "sdd", "sde", "sdf"} {
host.DiskIO = append(host.DiskIO, models.DiskIO{Device: device, ReadBytes: 1 << 20, WriteBytes: 1 << 20, IOTime: 100})
}
- m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second))
+ m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second), counterSampleTime{})
for i := range host.DiskIO {
host.DiskIO[i].ReadBytes += 1 << 20
host.DiskIO[i].WriteBytes += 1 << 20
host.DiskIO[i].IOTime += 1000
}
- m.writeHostPhysicalDiskIOMetrics(host, now)
+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampleTime{})
m.writeHostSMARTMetrics(host, now)
store.Flush()
diff --git a/internal/monitoring/monitor_agents.go b/internal/monitoring/monitor_agents.go
index ff42a8a809..ed04ec56f0 100644
--- a/internal/monitoring/monitor_agents.go
+++ b/internal/monitoring/monitor_agents.go
@@ -226,6 +226,46 @@ func advanceHostReportOrder(
return order, accepted
}
+// agentAuthoredCounterClock names the reporting machine's own clock, on which
+// an agent authors each report's collection time.
+const agentAuthoredCounterClock = "agent-authored"
+
+// agentCounterSampleTime returns when an accepted agent report's cumulative
+// counters were collected. The stream watermark admits sequenced reports in
+// order, so the authored times of consecutive accepted reports measure the
+// real collection interval, across an agent restart on the same machine clock
+// too, even when reconnect delivery drains a buffered FIFO back to back and
+// Pulse receipt times collapse toward zero. Legacy and untimestamped reports
+// have no ordered source clock and keep receipt time; a switch between the two
+// starts a new counter baseline.
+func agentCounterSampleTime(sequenceID string, authoredAt time.Time) counterSampleTime {
+ if _, _, sequenced := agentshost.ParseReportSequenceID(sequenceID); !sequenced || authoredAt.IsZero() {
+ return counterSampleTime{}
+ }
+ return counterSampleTime{clock: agentAuthoredCounterClock, at: authoredAt.UTC()}
+}
+
+// agentIOCounterReadClock is the sampling mode of a host report whose agent
+// stamped when it read the DiskIO and Network counters. It shares the machine
+// clock with report Timestamp but not its offset from the read, so a switch
+// between the two modes starts a new counter baseline.
+const agentIOCounterReadClock = "agent-io-counter-read"
+
+// agentIOCounterSampleTime returns when an accepted host report's DiskIO and
+// Network counters were read. A current agent stamps the read itself, and the
+// report Timestamp follows every optional collector, so the stamp is preferred.
+// Older agents, and a missing or zero stamp, keep the report-Timestamp
+// fallback, and legacy reports keep receipt time.
+func agentIOCounterSampleTime(report agentshost.Report) counterSampleTime {
+ if report.IOCountersCollectedAt != nil {
+ if sampled := agentCounterSampleTime(report.SequenceID, *report.IOCountersCollectedAt); sampled.clock != "" {
+ sampled.clock = agentIOCounterReadClock
+ return sampled
+ }
+ }
+ return agentCounterSampleTime(report.SequenceID, report.Timestamp)
+}
+
// lockHostReportApplication serializes the complete state transition for one
// host while allowing different hosts to ingest concurrently. Reserving a
// sequence number without holding this lock through the subsequent state write
@@ -2350,8 +2390,12 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
dockerVersion = runtimeVersion
}
+ counterSampled := agentCounterSampleTime(report.SequenceID, report.Timestamp)
containers := make([]models.DockerContainer, 0, len(report.Containers))
- for _, payload := range report.Containers {
+ // Legacy container fields remain numeric. Carry the current calculation's
+ // validity through to history so warmup and clock faults cannot become idle.
+ containerNetworkValidity := make([]models.IORateValidity, len(report.Containers))
+ for i, payload := range report.Containers {
container := models.DockerContainer{
ID: payload.ID,
Name: payload.Name,
@@ -2451,9 +2495,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
metrics.DiskWrite = clampToInt64(payload.BlockIO.WriteBytes)
}
- readRate, writeRate, netInRate, netOutRate := m.rateTracker.CalculateRates(
+ readRate, writeRate, _, netInRate, netOutRate := m.rateTracker.calculateSampledRates(
fmt.Sprintf("docker:%s:%s", identifier, containerIdentifier),
metrics,
+ counterSampled,
)
if container.BlockIO != nil && readRate >= 0 {
@@ -2466,9 +2511,11 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
}
if netInRate >= 0 {
container.NetInRate = netInRate
+ containerNetworkValidity[i].NetworkIn = true
}
if netOutRate >= 0 {
container.NetOutRate = netOutRate
+ containerNetworkValidity[i].NetworkOut = true
}
}
@@ -2722,7 +2769,7 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
}
// Use a prefixed key (docker:containerID) to distinguish from Proxmox containers.
- for _, container := range containers {
+ for i, container := range containers {
if container.ID == "" {
continue
}
@@ -2743,10 +2790,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
if m.metricsHistory != nil {
m.metricsHistory.AddGuestMetric(metricKey, "cpu", models.DockerContainerCPUCapacityPercent(container, host.CPUs), now)
m.metricsHistory.AddGuestMetric(metricKey, "memory", container.MemoryPercent, now)
- if container.NetInRate >= 0 {
+ if containerNetworkValidity[i].NetworkIn {
m.metricsHistory.AddGuestMetric(metricKey, "netin", container.NetInRate, now)
}
- if container.NetOutRate >= 0 {
+ if containerNetworkValidity[i].NetworkOut {
m.metricsHistory.AddGuestMetric(metricKey, "netout", container.NetOutRate, now)
}
if diskReadRate >= 0 {
@@ -2760,10 +2807,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con
if m.metricsStore != nil {
m.metricsStore.Write("dockerContainer", container.ID, "cpu", models.DockerContainerCPUCapacityPercent(container, host.CPUs), now)
m.metricsStore.Write("dockerContainer", container.ID, "memory", container.MemoryPercent, now)
- if container.NetInRate >= 0 {
+ if containerNetworkValidity[i].NetworkIn {
m.metricsStore.Write("dockerContainer", container.ID, "netin", container.NetInRate, now)
}
- if container.NetOutRate >= 0 {
+ if containerNetworkValidity[i].NetworkOut {
m.metricsStore.Write("dockerContainer", container.ID, "netout", container.NetOutRate, now)
}
if diskReadRate >= 0 {
@@ -3542,6 +3589,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.
identifier,
previousHostModel,
observedAt,
+ agentLibvirtSampleTime(report.SequenceID, report.Libvirt),
)
xcpngData := normalizeAgentXCPNGInventory(report.XCPNG, previousHostModel, observedAt)
@@ -3744,6 +3792,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.
// Network and disk bytes from the agent are cumulative totals since boot;
// the RateTracker converts them to bytes/second, just like VMs and containers.
now := receivedAt
+ counterSampled := agentIOCounterSampleTime(report)
var totalRXBytes, totalTXBytes uint64
for _, nic := range host.NetworkInterfaces {
@@ -3767,7 +3816,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.
NetworkOut: int64(totalTXBytes),
Timestamp: now,
}
- diskReadRate, diskWriteRate, netInRate, netOutRate := m.rateTracker.CalculateRates(hostRateKey, currentMetrics)
+ diskReadRate, diskWriteRate, _, netInRate, netOutRate := m.rateTracker.calculateSampledRates(hostRateKey, currentMetrics, counterSampled)
// Store computed rates on the host model so they flow through to unified resources
if netInRate >= 0 {
@@ -3878,7 +3927,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.
}
}
- m.writeHostPhysicalDiskIOMetrics(host, now)
+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampled)
if m.metricsStore != nil {
m.metricsStore.Write("agent", host.ID, "cpu", host.CPUUsage, now)
@@ -4398,7 +4447,10 @@ func hostDiskIOMetricResourceID(host models.Host, io models.DiskIO, proxmoxDisks
return fmt.Sprintf("%s:%s", strings.TrimSpace(host.ID), device)
}
-func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time) {
+// writeHostPhysicalDiskIOMetrics records per-device disk rates. now is Pulse
+// receipt time for history and rate-series freshness; sampled times the
+// counter interval itself.
+func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time, sampled counterSampleTime) {
if shouldSkipNativeMockStateMetricWrites() || (m.metricsHistory == nil && m.metricsStore == nil) {
return
}
@@ -4429,7 +4481,7 @@ func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time
DiskBusy: int64(io.IOTime),
Timestamp: now,
}
- readRate, writeRate, busyPct, _, _ := m.rateTracker.CalculateRatesWithBusy(trackerKey, current)
+ readRate, writeRate, busyPct, _, _ := m.rateTracker.calculateSampledRates(trackerKey, current, sampled)
if readRate >= 0 {
if m.metricsHistory != nil {
diff --git a/internal/monitoring/monitor_libvirt.go b/internal/monitoring/monitor_libvirt.go
index 6dd306d60d..2ca0f30363 100644
--- a/internal/monitoring/monitor_libvirt.go
+++ b/internal/monitoring/monitor_libvirt.go
@@ -18,11 +18,28 @@ const (
maxAgentLibvirtDomainNameBytes = 256
)
+// agentLibvirtSampleTime returns when a libvirt inventory's domain counters
+// were sampled. The agent stamps CollectedAt on its own clock just before it
+// reads the domain stats, nearer that sample than the report's
+// end-of-collection Timestamp. As with agentCounterSampleTime, only a
+// sequenced report is ordered on that clock; legacy reports and inventories
+// without CollectedAt keep receipt time.
+func agentLibvirtSampleTime(sequenceID string, inventory *agentshost.LibvirtInventory) counterSampleTime {
+ if inventory == nil {
+ return counterSampleTime{}
+ }
+ return agentCounterSampleTime(sequenceID, inventory.CollectedAt)
+}
+
+// normalizeAgentLibvirtInventory admits a host report's libvirt inventory.
+// observedAt is Pulse receipt time for freshness; sampled times the CPU and
+// I/O counter intervals.
func (m *Monitor) normalizeAgentLibvirtInventory(
report *agentshost.LibvirtInventory,
hostID string,
previous *models.Host,
observedAt time.Time,
+ sampled counterSampleTime,
) *models.HostLibvirtInventory {
if report == nil {
if previous != nil && previous.Libvirt != nil {
@@ -34,13 +51,9 @@ func (m *Monitor) normalizeAgentLibvirtInventory(
}
previousByID := make(map[string]models.HostLibvirtDomain)
- previousAt := time.Time{}
- if previous != nil {
- previousAt = previous.LastSeen
- if previous.Libvirt != nil {
- for _, domain := range previous.Libvirt.Domains {
- previousByID[domain.ID] = domain
- }
+ if previous != nil && previous.Libvirt != nil {
+ for _, domain := range previous.Libvirt.Domains {
+ previousByID[domain.ID] = domain
}
}
@@ -80,17 +93,24 @@ func (m *Monitor) normalizeAgentLibvirtInventory(
domain.MemoryCurrentBytes = domain.MemoryMaximumBytes
}
- if prior, ok := previousByID[domain.ID]; ok {
- domain.CPUUsagePercent, domain.CPUUsageValid = libvirtCPUUsagePercent(
- prior,
- domain,
- observedAt.Sub(previousAt),
+ if m != nil && m.rateTracker != nil {
+ seriesID := "libvirt:" + strings.TrimSpace(hostID) + ":" + domain.ID
+ // CPU time is sampled with the domain's I/O counters, so it is
+ // measured over the same interval on the same clock. Every sample
+ // moves its baseline; only a domain present in the previous report
+ // whose CPU time did not reset yields a usage.
+ cpuRate := m.rateTracker.calculateSampledCounterRate(
+ seriesID+":cpu",
+ domain.CPUTimeNanoseconds,
+ observedAt,
+ sampled,
)
- }
+ if prior, ok := previousByID[domain.ID]; ok && domain.CPUTimeNanoseconds >= prior.CPUTimeNanoseconds {
+ domain.CPUUsagePercent, domain.CPUUsageValid = libvirtCPUUsagePercent(cpuRate, domain.VCPUs)
+ }
- if m != nil && m.rateTracker != nil {
- diskRead, diskWrite, netIn, netOut := m.rateTracker.CalculateRates(
- "libvirt:"+strings.TrimSpace(hostID)+":"+domain.ID,
+ diskRead, diskWrite, _, netIn, netOut := m.rateTracker.calculateSampledRates(
+ seriesID,
IOMetrics{
DiskRead: uint64ToInt64(domain.DiskReadBytes),
DiskWrite: uint64ToInt64(domain.DiskWriteBytes),
@@ -98,6 +118,7 @@ func (m *Monitor) normalizeAgentLibvirtInventory(
NetworkOut: uint64ToInt64(domain.NetworkTXBytes),
Timestamp: observedAt,
},
+ sampled,
)
if diskRead >= 0 && diskWrite >= 0 && netIn >= 0 && netOut >= 0 {
domain.DiskReadRate = diskRead
@@ -142,16 +163,17 @@ func agentLibvirtDomainID(name string) string {
return "domain-" + hex.EncodeToString(sum[:8])
}
-func libvirtCPUUsagePercent(previous, current models.HostLibvirtDomain, elapsed time.Duration) (float64, bool) {
- if elapsed <= 0 || current.CPUTimeNanoseconds < previous.CPUTimeNanoseconds {
+// libvirtCPUUsagePercent converts a domain's CPU-time rate, in nanoseconds per
+// second summed across its vCPUs, to a percentage of its vCPU capacity. A
+// negative rate means no comparable interval.
+func libvirtCPUUsagePercent(cpuNanosecondsPerSecond float64, vcpus int) (float64, bool) {
+ if cpuNanosecondsPerSecond < 0 {
return 0, false
}
- vcpus := current.VCPUs
if vcpus <= 0 {
vcpus = 1
}
- delta := current.CPUTimeNanoseconds - previous.CPUTimeNanoseconds
- percent := (float64(delta) / float64(elapsed.Nanoseconds()) / float64(vcpus)) * 100
+ percent := (cpuNanosecondsPerSecond / float64(time.Second) / float64(vcpus)) * 100
if math.IsNaN(percent) || math.IsInf(percent, 0) {
return 0, false
}
diff --git a/internal/monitoring/monitor_libvirt_test.go b/internal/monitoring/monitor_libvirt_test.go
index cbf79becc8..ebc5c072b1 100644
--- a/internal/monitoring/monitor_libvirt_test.go
+++ b/internal/monitoring/monitor_libvirt_test.go
@@ -24,7 +24,7 @@ func TestNormalizeAgentLibvirtInventoryDerivesCPUAndIORates(t *testing.T) {
DiskReadBytes: 3000,
DiskWriteBytes: 4000,
}}}
- first := monitor.normalizeAgentLibvirtInventory(firstReport, "host-1", nil, t0)
+ first := monitor.normalizeAgentLibvirtInventory(firstReport, "host-1", nil, t0, counterSampleTime{})
if first == nil || len(first.Domains) != 1 {
t.Fatalf("first inventory = %+v", first)
}
@@ -49,7 +49,7 @@ func TestNormalizeAgentLibvirtInventoryDerivesCPUAndIORates(t *testing.T) {
DiskReadBytes: 6000,
DiskWriteBytes: 8000,
}}}
- second := monitor.normalizeAgentLibvirtInventory(secondReport, "host-1", previous, t0.Add(10*time.Second))
+ second := monitor.normalizeAgentLibvirtInventory(secondReport, "host-1", previous, t0.Add(10*time.Second), counterSampleTime{})
if second == nil || len(second.Domains) != 1 {
t.Fatalf("second inventory = %+v", second)
}
@@ -76,7 +76,7 @@ func TestNormalizeAgentLibvirtInventoryBoundsAndDeduplicates(t *testing.T) {
{Name: "vm", State: "running"},
{Name: "bad\x00name", State: "running"},
}}
- got := monitor.normalizeAgentLibvirtInventory(report, "host-1", nil, time.Now())
+ got := monitor.normalizeAgentLibvirtInventory(report, "host-1", nil, time.Now(), counterSampleTime{})
if got == nil || len(got.Domains) != 1 {
t.Fatalf("inventory = %+v, want one domain", got)
}
@@ -99,6 +99,7 @@ func TestNormalizeAgentLibvirtInventoryPreservesLastSuccessfulSampleOnFailure(t
"host-1",
previous,
time.Now(),
+ counterSampleTime{},
)
if got == nil || len(got.Domains) != 1 || !got.CollectedAt.Equal(collectedAt) {
t.Fatalf("preserved inventory = %+v", got)
diff --git a/internal/monitoring/ratetracker.go b/internal/monitoring/ratetracker.go
index 13a59cd8c2..3c5ecbfe94 100644
--- a/internal/monitoring/ratetracker.go
+++ b/internal/monitoring/ratetracker.go
@@ -10,20 +10,41 @@ import (
// IOMetrics is an alias for models.IOMetrics.
type IOMetrics = models.IOMetrics
-type counterBaseline struct {
- value int64
+type counterBaseline[T int64 | uint64] struct {
+ value T
observedAt time.Time
initialized bool
}
type counterHistory struct {
- diskRead counterBaseline
- diskWrite counterBaseline
- diskBusy counterBaseline
- networkIn counterBaseline
- networkOut counterBaseline
+ diskRead counterBaseline[int64]
+ diskWrite counterBaseline[int64]
+ diskBusy counterBaseline[int64]
+ networkIn counterBaseline[int64]
+ networkOut counterBaseline[int64]
+ cpuTime counterBaseline[uint64]
lastObservedAt time.Time
sourceUptime uint64
+ sampleClock string
+}
+
+// counterSampleTime is when a producer sampled a set of cumulative counters,
+// on a named clock whose samples the producer has already ordered. Rates
+// measure intervals on that clock, while IOMetrics.Timestamp stays Pulse
+// receipt time for freshness and cleanup. The zero value keeps receipt time
+// for both, which pull-based collectors observe directly.
+type counterSampleTime struct {
+ clock string
+ at time.Time
+}
+
+// counterTime returns when a counter was sampled: the named clock's time, or
+// else the counter's own receipt time with the payload receipt as fallback.
+func (sampled counterSampleTime) counterTime(counterReceivedAt, receivedAt time.Time) time.Time {
+ if sampled.clock != "" {
+ return sampled.at
+ }
+ return observationTime(counterReceivedAt, receivedAt)
}
// RateTracker converts cumulative byte counters into adjacent-sample rates.
@@ -41,8 +62,9 @@ func NewRateTracker() *RateTracker {
// CalculateRates calculates disk and network rates in bytes per second.
// A negative result means the upstream counter was absent, the sample was
-// out-of-order, or no earlier observation exists. A returned zero is a valid
-// observed idle/reset interval.
+// out-of-order, no earlier observation exists, or no comparable interval
+// exists on the sample's clock. A returned zero is a valid observed
+// idle/reset interval.
func (rt *RateTracker) CalculateRates(guestID string, current IOMetrics) (diskReadRate, diskWriteRate, netInRate, netOutRate float64) {
diskReadRate, diskWriteRate, _, netInRate, netOutRate = rt.calculateRates(guestID, current)
return
@@ -55,18 +77,20 @@ func (rt *RateTracker) CalculateRatesWithBusy(guestID string, current IOMetrics)
}
func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskReadRate, diskWriteRate, diskBusyPct, netInRate, netOutRate float64) {
+ return rt.calculateSampledRates(guestID, current, counterSampleTime{})
+}
+
+// calculateSampledRates measures each counter interval on sampled's clock.
+// Samples on a different clock than the baseline start a new counter epoch,
+// because their times cannot be subtracted from each other.
+func (rt *RateTracker) calculateSampledRates(guestID string, current IOMetrics, sampled counterSampleTime) (diskReadRate, diskWriteRate, diskBusyPct, netInRate, netOutRate float64) {
rt.mu.Lock()
defer rt.mu.Unlock()
if current.Timestamp.IsZero() {
return -1, -1, -1, -1, -1
}
-
- history := rt.history[guestID]
- if history == nil {
- history = &counterHistory{}
- rt.history[guestID] = history
- }
+ history, sampled := rt.sampledHistory(guestID, current.Timestamp, sampled)
if current.SourceUptime > 0 {
if history.sourceUptime > 0 && current.SourceUptime < history.sourceUptime {
history.resetCounterEpoch()
@@ -75,11 +99,12 @@ func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskRe
}
presence := current.Presence.Effective()
- diskReadRate = calculateCounterRate(&history.diskRead, current.DiskRead, observationTime(current.ObservedAt.DiskRead, current.Timestamp), presence.DiskRead)
- diskWriteRate = calculateCounterRate(&history.diskWrite, current.DiskWrite, observationTime(current.ObservedAt.DiskWrite, current.Timestamp), presence.DiskWrite)
- diskBusyRate := calculateCounterRate(&history.diskBusy, current.DiskBusy, observationTime(current.ObservedAt.DiskBusy, current.Timestamp), presence.DiskBusy)
- netInRate = calculateCounterRate(&history.networkIn, current.NetworkIn, observationTime(current.ObservedAt.NetworkIn, current.Timestamp), presence.NetworkIn)
- netOutRate = calculateCounterRate(&history.networkOut, current.NetworkOut, observationTime(current.ObservedAt.NetworkOut, current.Timestamp), presence.NetworkOut)
+ ordered := sampled.clock != ""
+ diskReadRate = sampledCounterRate(&history.diskRead, current.DiskRead, sampled.counterTime(current.ObservedAt.DiskRead, current.Timestamp), presence.DiskRead, ordered)
+ diskWriteRate = sampledCounterRate(&history.diskWrite, current.DiskWrite, sampled.counterTime(current.ObservedAt.DiskWrite, current.Timestamp), presence.DiskWrite, ordered)
+ diskBusyRate := sampledCounterRate(&history.diskBusy, current.DiskBusy, sampled.counterTime(current.ObservedAt.DiskBusy, current.Timestamp), presence.DiskBusy, ordered)
+ netInRate = sampledCounterRate(&history.networkIn, current.NetworkIn, sampled.counterTime(current.ObservedAt.NetworkIn, current.Timestamp), presence.NetworkIn, ordered)
+ netOutRate = sampledCounterRate(&history.networkOut, current.NetworkOut, sampled.counterTime(current.ObservedAt.NetworkOut, current.Timestamp), presence.NetworkOut, ordered)
if diskBusyRate < 0 {
diskBusyPct = -1
@@ -92,18 +117,51 @@ func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskRe
}
}
- if current.Timestamp.After(history.lastObservedAt) {
- history.lastObservedAt = current.Timestamp
- }
return
}
+// sampledHistory applies clock epochs and receipt-time cleanup to every
+// counter series. The caller holds rt.mu and supplies a nonzero receipt time.
+func (rt *RateTracker) sampledHistory(seriesID string, receivedAt time.Time, sampled counterSampleTime) (*counterHistory, counterSampleTime) {
+ if sampled.at.IsZero() {
+ sampled = counterSampleTime{}
+ }
+ history := rt.history[seriesID]
+ if history == nil {
+ history = &counterHistory{}
+ rt.history[seriesID] = history
+ }
+ if sampled.clock != history.sampleClock {
+ history.resetCounterEpoch()
+ history.sampleClock = sampled.clock
+ }
+ if receivedAt.After(history.lastObservedAt) {
+ history.lastObservedAt = receivedAt
+ }
+ return history, sampled
+}
+
+// calculateSampledCounterRate measures one cumulative counter outside the disk
+// and network set, such as a libvirt domain's CPU time, under the same clock,
+// baseline and cleanup rules as calculateSampledRates. Preserve its unsigned
+// range: cumulative CPU nanoseconds can exceed the signed byte-counter range.
+func (rt *RateTracker) calculateSampledCounterRate(seriesID string, value uint64, receivedAt time.Time, sampled counterSampleTime) float64 {
+ rt.mu.Lock()
+ defer rt.mu.Unlock()
+ if receivedAt.IsZero() {
+ return -1
+ }
+ history, sampled := rt.sampledHistory(seriesID, receivedAt, sampled)
+ return sampledCounterRate(&history.cpuTime, value, sampled.counterTime(time.Time{}, receivedAt), true, sampled.clock != "")
+}
+
func (history *counterHistory) resetCounterEpoch() {
- history.diskRead = counterBaseline{}
- history.diskWrite = counterBaseline{}
- history.diskBusy = counterBaseline{}
- history.networkIn = counterBaseline{}
- history.networkOut = counterBaseline{}
+ history.diskRead = counterBaseline[int64]{}
+ history.diskWrite = counterBaseline[int64]{}
+ history.diskBusy = counterBaseline[int64]{}
+ history.networkIn = counterBaseline[int64]{}
+ history.networkOut = counterBaseline[int64]{}
+ history.cpuTime = counterBaseline[uint64]{}
}
func observationTime(counterTime, fallback time.Time) time.Time {
@@ -113,7 +171,7 @@ func observationTime(counterTime, fallback time.Time) time.Time {
return counterTime
}
-func calculateCounterRate(baseline *counterBaseline, value int64, observedAt time.Time, present bool) float64 {
+func calculateCounterRate[T int64 | uint64](baseline *counterBaseline[T], value T, observedAt time.Time, present bool) float64 {
if !present {
return -1
}
@@ -140,6 +198,19 @@ func calculateCounterRate(baseline *counterBaseline, value int64, observedAt tim
return float64(value-previous) / elapsed
}
+// sampledCounterRate is calculateCounterRate for a sample that may sit on an
+// ordered clock. There a time that does not advance past the baseline is a
+// clock step or a malformed time, not a late sample, so neither sample bounds
+// a known interval: the baseline is dropped, this sample is unknown, and the
+// next sample starts a new baseline.
+func sampledCounterRate[T int64 | uint64](baseline *counterBaseline[T], value T, observedAt time.Time, present, ordered bool) float64 {
+ if ordered && present && baseline.initialized && !observedAt.After(baseline.observedAt) {
+ *baseline = counterBaseline[T]{}
+ return -1
+ }
+ return calculateCounterRate(baseline, value, observedAt, present)
+}
+
// Clear removes all tracked data.
func (rt *RateTracker) Clear() {
rt.mu.Lock()
diff --git a/pkg/agents/host/report.go b/pkg/agents/host/report.go
index e9bab8028d..64b7486268 100644
--- a/pkg/agents/host/report.go
+++ b/pkg/agents/host/report.go
@@ -65,6 +65,11 @@ type Report struct {
Tags []string `json:"tags,omitempty"`
Timestamp time.Time `json:"timestamp"`
SequenceID string `json:"sequenceId,omitempty"`
+ // IOCountersCollectedAt is when the agent read the cumulative DiskIO and
+ // Network counters, on the same clock as Timestamp. Timestamp is stamped
+ // after every optional collector, so only this bounds the counter read.
+ // Optional: older agents omit it.
+ IOCountersCollectedAt *time.Time `json:"ioCountersCollectedAt,omitempty"`
}
// LibvirtInventory is the bounded, read-only domain inventory collected from
diff --git a/pkg/agents/host/report_io_counters_collected_at_test.go b/pkg/agents/host/report_io_counters_collected_at_test.go
new file mode 100644
index 0000000000..9faa966bf4
--- /dev/null
+++ b/pkg/agents/host/report_io_counters_collected_at_test.go
@@ -0,0 +1,47 @@
+package host
+
+import (
+ "bytes"
+ "encoding/json"
+ "testing"
+ "time"
+)
+
+// IOCountersCollectedAt is optional on the wire: an older agent's report
+// decodes without it, a report without it emits nothing new, and a stamped
+// report keeps it apart from Timestamp through the JSON that both delivery
+// and the agent's persisted report buffer use.
+func TestReportIOCountersCollectedAtIsOptionalOnTheWire(t *testing.T) {
+ var older Report
+ if err := json.Unmarshal([]byte(`{"agent":{"id":"agent-a"},"host":{"hostname":"host-a"},"timestamp":"2026-10-30T12:00:50Z","sequenceId":"v1:stream-a:2"}`), &older); err != nil {
+ t.Fatalf("decode older report: %v", err)
+ }
+ if older.IOCountersCollectedAt != nil {
+ t.Fatalf("older report decoded a counter read stamp: %v", older.IOCountersCollectedAt)
+ }
+
+ unstamped, err := json.Marshal(Report{Timestamp: time.Date(2026, 10, 30, 12, 0, 50, 0, time.UTC)})
+ if err != nil {
+ t.Fatalf("encode unstamped report: %v", err)
+ }
+ if bytes.Contains(unstamped, []byte("ioCountersCollectedAt")) {
+ t.Fatalf("unstamped report emitted the optional field: %s", unstamped)
+ }
+
+ readAt := time.Date(2026, 10, 30, 12, 0, 30, 123456789, time.UTC)
+ stamped := Report{Timestamp: readAt.Add(20 * time.Second), IOCountersCollectedAt: &readAt}
+ raw, err := json.Marshal(stamped)
+ if err != nil {
+ t.Fatalf("encode stamped report: %v", err)
+ }
+ var decoded Report
+ if err := json.Unmarshal(raw, &decoded); err != nil {
+ t.Fatalf("decode stamped report: %v", err)
+ }
+ if decoded.IOCountersCollectedAt == nil || !decoded.IOCountersCollectedAt.Equal(readAt) {
+ t.Fatalf("counter read stamp = %v, want %v", decoded.IOCountersCollectedAt, readAt)
+ }
+ if !decoded.Timestamp.Equal(stamped.Timestamp) {
+ t.Fatalf("Timestamp = %v, want the report's own %v", decoded.Timestamp, stamped.Timestamp)
+ }
+}
diff --git a/scripts/release_control/canonical_completion_guard_test.py b/scripts/release_control/canonical_completion_guard_test.py
index 040d162608..632e3a7360 100644
--- a/scripts/release_control/canonical_completion_guard_test.py
+++ b/scripts/release_control/canonical_completion_guard_test.py
@@ -256,9 +256,13 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"allow_same_subsystem_tests": False,
"test_prefixes": [],
"exact_files": [
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
"internal/config/host_continuity_test.go",
"internal/models/metrics_types_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/availability_probe_agent_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
@@ -447,9 +451,13 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"allow_same_subsystem_tests": False,
"test_prefixes": [],
"exact_files": [
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
"internal/config/host_continuity_test.go",
"internal/models/metrics_types_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/availability_probe_agent_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
@@ -486,6 +494,9 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"test_prefixes": [],
"exact_files": [
"internal/config/host_continuity_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
"internal/monitoring/issue1485_unraid_lifecycle_test.go",
@@ -495,6 +506,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_package_updates_test.go",
"internal/unifiedresources/code_standards_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go",
],
}
@@ -776,6 +788,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"test_prefixes": [],
"exact_files": [
"internal/agentupdate/coverage_test.go",
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/hostagent/action_runner_client_test.go",
"internal/hostagent/agent_flushbuffer_test.go",
"internal/hostagent/agent_metrics_test.go",
@@ -795,6 +808,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"internal/hostagent/smartctl_standby_guard_test.go",
"internal/hostagent/storage_cleanup_test.go",
"internal/hostagent/unraid_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go",
],
}
diff --git a/scripts/release_control/subsystem_lookup_test.py b/scripts/release_control/subsystem_lookup_test.py
index 61b94eb0f1..847cab70da 100644
--- a/scripts/release_control/subsystem_lookup_test.py
+++ b/scripts/release_control/subsystem_lookup_test.py
@@ -4644,9 +4644,13 @@ class SubsystemLookupTest(unittest.TestCase):
self.assertEqual(
lifecycle_match["verification_requirement"]["exact_files"],
[
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
"internal/config/host_continuity_test.go",
"internal/models/metrics_types_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/availability_probe_agent_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
@@ -4675,6 +4679,9 @@ class SubsystemLookupTest(unittest.TestCase):
monitoring_match["verification_requirement"]["exact_files"],
[
"internal/config/host_continuity_test.go",
+ "internal/monitoring/agent_report_counter_clock_test.go",
+ "internal/monitoring/agent_report_io_counter_clock_test.go",
+ "internal/monitoring/agent_report_libvirt_clock_test.go",
"internal/monitoring/docker_filesystem_evidence_test.go",
"internal/monitoring/docker_metric_presence_test.go",
"internal/monitoring/issue1485_unraid_lifecycle_test.go",
@@ -4684,6 +4691,7 @@ class SubsystemLookupTest(unittest.TestCase):
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_package_updates_test.go",
"internal/unifiedresources/code_standards_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go",
],
)
@@ -4989,6 +4997,7 @@ class SubsystemLookupTest(unittest.TestCase):
match["verification_requirement"]["exact_files"],
[
"internal/agentupdate/coverage_test.go",
+ "internal/api/agent_report_decode_io_stamp_compat_test.go",
"internal/hostagent/action_runner_client_test.go",
"internal/hostagent/agent_flushbuffer_test.go",
"internal/hostagent/agent_metrics_test.go",
@@ -5008,6 +5017,7 @@ class SubsystemLookupTest(unittest.TestCase):
"internal/hostagent/smartctl_standby_guard_test.go",
"internal/hostagent/storage_cleanup_test.go",
"internal/hostagent/unraid_test.go",
+ "pkg/agents/host/report_io_counters_collected_at_test.go",
"pkg/agents/host/report_test.go",
],
)