Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
fix(monitoring): use collection clocks for agent counter rates
rcourtman6dd202276c1d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 0a3932108e..653ee593a4 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -2341,7 +2341,10 @@ update, profile rollout, command reachability, or fleet-control authority.28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.Every current agent process authors a process-unique report stream and a- monotonic sequence within that stream. Reconnect delivery must preserve+ monotonic sequence within that stream. The optional+ `ioCountersCollectedAt` records when the metrics snapshot read the+ cumulative disk-I/O and network counters; `timestamp` stays the report's+ end-of-collection time for age and order. Reconnect delivery must preservecollection order by draining each destination's persisted FIFO beforesending a newly collected report, so buffered running-task snapshots cannotoverwrite a later cancellation or completion. Authentication/scopediff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex c0897f0364..e9fcb7a166 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -3505,6 +3505,34 @@ buffered complete inventory overwrite a newer helper-degraded status report.Legacy reports without a sequence retain timestamp-based reconnect-burstprotection, but a clock correction after a normal report interval must beadmitted rather than freezing telemetry indefinitely.+Counter rates follow the same ownership. An accepted sequenced report that+carries its authored time is measured on the reporting machine's clock: host+network and disk, physical-disk and container rates divide by the interval+between the authored collection times of consecutive accepted reports, across+agent restarts too, so a FIFO drained back to back cannot compress collection+intervals into milliseconds of receipt time and fabricate throughput spikes.+A current host agent also sends `ioCountersCollectedAt`, stamped right after+the metrics snapshot reads its disk-I/O and network counters. Host and+physical-disk rates are measured on it rather than on the report `timestamp`,+which follows every optional collector, and a switch between the two sampling+modes starts a new counter baseline.+Libvirt domain CPU and I/O counters inside an accepted host report are measured+on the inventory's `CollectedAt`, which the agent stamps on the same clock just+before it reads the domain stats, and CPU usage shares the I/O interval rather+than the gap since the previous host report.+Legacy and untimestamped reports keep receipt-time rates. A switch between the+two clocks starts a new counter baseline, and an authored time that does not+advance past the baseline, a backward step or a repeated timestamp, drops it,+so no rate is reported until two samples bound a new interval. Liveness,+history timestamps and rate-series cleanup keep Pulse receipt time.+Unknown Docker network intervals must be omitted from both history writers;+the numeric compatibility fields must not turn first samples or clock faults+into measured idle. A genuine unchanged-counter interval still writes zero.+Libvirt CPU counter baselines retain the full unsigned nanosecond range rather+than clamping accumulated CPU time to a signed byte-counter value. The+`agent_report_counter_clock_test.go`, `agent_report_io_counter_clock_test.go`+and `agent_report_libvirt_clock_test.go` regressions cover reconnect delivery,+clock transitions, idle/reset intervals and counter range preservation.Host telemetry also has a reporting lease derived from the agent cadence.Monitoring must not clear a genuinely active Unraid parity operation or LinuxRAID rebuild during an ordinary polling gap. Once the lease expires, it mustdiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex 00863a1873..5c60f4a468 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -1547,9 +1547,13 @@"allow_same_subsystem_tests": false,"test_prefixes": [],"exact_files": [+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/api/host_agent_removal_lifecycle_integration_test.go","internal/config/host_continuity_test.go","internal/models/metrics_types_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/availability_probe_agent_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go",@@ -1654,7 +1658,8 @@"test_prefixes": [],"exact_files": ["internal/hostmetrics/collector_stuck_mount_test.go",- "internal/hostmetrics/collector_test.go"+ "internal/hostmetrics/collector_test.go",+ "internal/hostmetrics/io_counters_collected_at_test.go"]},{@@ -1670,6 +1675,7 @@"test_prefixes": [],"exact_files": ["internal/agentupdate/coverage_test.go",+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/hostagent/action_runner_client_test.go","internal/hostagent/agent_flushbuffer_test.go","internal/hostagent/agent_metrics_test.go",@@ -1689,6 +1695,7 @@"internal/hostagent/smartctl_standby_guard_test.go","internal/hostagent/storage_cleanup_test.go","internal/hostagent/unraid_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go"]},@@ -6590,6 +6597,9 @@"test_prefixes": [],"exact_files": ["internal/config/host_continuity_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go","internal/monitoring/issue1485_unraid_lifecycle_test.go",@@ -6599,6 +6609,7 @@"internal/monitoring/monitor_host_agents_test.go","internal/monitoring/monitor_package_updates_test.go","internal/unifiedresources/code_standards_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go"]},@@ -7060,6 +7071,9 @@"internal/config/docker_metadata_test.go","internal/config/guest_metadata_test.go","internal/config/guest_metadata_writer_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/availability_poller_test.go","internal/monitoring/availability_probe_agent_test.go","internal/monitoring/availability_udp_test.go",diff --git a/internal/api/agent_report_decode_io_stamp_compat_test.go b/internal/api/agent_report_decode_io_stamp_compat_test.gonew file mode 100644index 0000000000..be16110f02--- /dev/null+++ b/internal/api/agent_report_decode_io_stamp_compat_test.go@@ -0,0 +1,91 @@+package api++import (+ "bytes"+ "compress/gzip"+ "net/http"+ "net/http/httptest"+ "testing"+ "time"+)++// preIOStampHostReport is how a receiver built before ioCountersCollectedAt+// existed sees a host report: the identity, ordering and counter fields it+// already relies on, and no counter read stamp.+type preIOStampHostReport struct {+ Agent struct {+ ID string `json:"id"`+ IntervalSeconds int `json:"intervalSeconds"`+ } `json:"agent"`+ Host struct {+ ID string `json:"id"`+ Hostname string `json:"hostname"`+ } `json:"host"`+ Network []struct {+ Name string `json:"name"`+ RXBytes uint64 `json:"rxBytes"`+ } `json:"network"`+ DiskIO []struct {+ Device string `json:"device"`+ ReadBytes uint64 `json:"readBytes"`+ } `json:"diskIO"`+ Timestamp time.Time `json:"timestamp"`+ SequenceID string `json:"sequenceId"`+}++const ioStampedHostReportJSON = `{"agent":{"id":"agent-a","intervalSeconds":30},` ++ `"host":{"id":"host-a","hostname":"host-a"},` ++ `"network":[{"name":"eth0","rxBytes":1000}],"diskIO":[{"device":"sda","readBytes":2000}],` ++ `"timestamp":"2026-10-30T12:00:50Z","sequenceId":"v1:stream-a:2",` ++ `"ioCountersCollectedAt":"2026-10-30T12:00:30.123456789Z"}`++// Preservation check: the report decoder accepts unknown fields, so a+// receiver without the optional counter read stamp still decodes a stamped+// report, plain or gzip, keeps every field it knows and ignores the stamp.+// It passes before and after the stamp is added.+func TestDecodeCompressedAgentReportIgnoresIOCounterStampOnPreChangeReceiver(t *testing.T) {+ var compressed bytes.Buffer+ writer := gzip.NewWriter(&compressed)+ if _, err := writer.Write([]byte(ioStampedHostReportJSON)); err != nil {+ t.Fatalf("gzip report: %v", err)+ }+ if err := writer.Close(); err != nil {+ t.Fatalf("close gzip writer: %v", err)+ }++ for _, tc := range []struct {+ name string+ body []byte+ encoding string+ }{+ {"uncompressed", []byte(ioStampedHostReportJSON), ""},+ {"gzip", compressed.Bytes(), "gzip"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ request := httptest.NewRequest(http.MethodPost, "/report", bytes.NewReader(tc.body))+ request.Header.Set("Content-Type", "application/json")+ if tc.encoding != "" {+ request.Header.Set("Content-Encoding", tc.encoding)+ }+ recorder := httptest.NewRecorder()++ var report preIOStampHostReport+ if !decodeCompressedAgentReport(recorder, request, 256*1024, 1536*1024, &report) {+ t.Fatalf("decoder refused a report carrying the optional stamp: %d %s", recorder.Code, recorder.Body.String())+ }+ if recorder.Body.Len() != 0 {+ t.Fatalf("decoder wrote a response on success: %s", recorder.Body.String())+ }+ wantTimestamp := time.Date(2026, 10, 30, 12, 0, 50, 0, time.UTC)+ if report.Agent.ID != "agent-a" || report.Agent.IntervalSeconds != 30 ||+ report.Host.ID != "host-a" || report.Host.Hostname != "host-a" ||+ !report.Timestamp.Equal(wantTimestamp) || report.SequenceID != "v1:stream-a:2" {+ t.Fatalf("known fields lost: %+v", report)+ }+ if len(report.Network) != 1 || report.Network[0].Name != "eth0" || report.Network[0].RXBytes != 1000 ||+ len(report.DiskIO) != 1 || report.DiskIO[0].Device != "sda" || report.DiskIO[0].ReadBytes != 2000 {+ t.Fatalf("counters lost: network %+v disk %+v", report.Network, report.DiskIO)+ }+ })+ }+}diff --git a/internal/hostagent/agent.go b/internal/hostagent/agent.goindex a23e76b78b..490d4ee307 100644--- a/internal/hostagent/agent.go+++ b/internal/hostagent/agent.go@@ -1448,6 +1448,9 @@ func (a *Agent) buildReport(ctx context.Context) (agentshost.Report, error) {Tags: append([]string(nil), runtimeConfig.tags...),Timestamp: a.collector.Now(),SequenceID: a.nextReportSequenceID(),+ // The counters were read inside the metrics snapshot, before the+ // optional collectors above; Timestamp only marks the report's end.+ IOCountersCollectedAt: snapshotIOCountersCollectedAt(snapshot),}return report, nildiff --git a/internal/hostagent/agent_metrics_test.go b/internal/hostagent/agent_metrics_test.goindex c457a1701e..de4c1be566 100644--- a/internal/hostagent/agent_metrics_test.go+++ b/internal/hostagent/agent_metrics_test.go@@ -95,7 +95,8 @@ func TestBuildReport(t *testing.T) {},metricsFn: func(ctx context.Context, diskExclude []string) (hostmetrics.Snapshot, error) {return hostmetrics.Snapshot{- CPUUsagePercent: 50.0,+ IOCountersCollectedAt: fixedTime.Add(-20 * time.Second),+ CPUUsagePercent: 50.0,Memory: agentshost.MemoryMetric{TotalBytes: 1000,UsedBytes: 500,@@ -179,6 +180,11 @@ func TestBuildReport(t *testing.T) {if !report.Timestamp.Equal(fixedTime) {t.Errorf("Timestamp = %v, want %v", report.Timestamp, fixedTime)}+ // The real report builder must carry the metrics snapshot's read time,+ // independently of the later end-of-collection timestamp.+ if report.IOCountersCollectedAt == nil || !report.IOCountersCollectedAt.Equal(fixedTime.Add(-20*time.Second)) {+ t.Fatalf("IOCountersCollectedAt = %v, want the earlier snapshot read", report.IOCountersCollectedAt)+ }})// Test case 2: Uptime failurediff --git a/internal/hostagent/sysinfo.go b/internal/hostagent/sysinfo.goindex 261c0c7274..e309c89a94 100644--- a/internal/hostagent/sysinfo.go+++ b/internal/hostagent/sysinfo.go@@ -55,6 +55,16 @@ func collectMetricsWithDiskFilters(ctx context.Context, collector SystemCollectoreturn collector.Metrics(ctx, exclude)}+// snapshotIOCountersCollectedAt carries the snapshot's counter read time onto+// the wire, absent when the collector did not stamp one.+func snapshotIOCountersCollectedAt(snapshot hostmetrics.Snapshot) *time.Time {+ if snapshot.IOCountersCollectedAt.IsZero() {+ return nil+ }+ collectedAt := snapshot.IOCountersCollectedAt.UTC()+ return &collectedAt+}+// NewDefaultCollector returns a SystemCollector that uses real OS calls.func NewDefaultCollector() SystemCollector {return &defaultCollector{}diff --git a/internal/hostmetrics/collector.go b/internal/hostmetrics/collector.goindex 4f7ee9c9d6..aa7396cff7 100644--- a/internal/hostmetrics/collector.go+++ b/internal/hostmetrics/collector.go@@ -33,6 +33,7 @@ var (diskIOCounters = godisk.IOCountersWithContextnetInterfaces = gonet.InterfacesWithContextnetIOCounters = gonet.IOCountersWithContext+ readZFSARCSize = readARCSize)// partitionsVisibleToAgent asks gopsutil to enumerate the mount namespace the@@ -153,8 +154,15 @@ type Snapshot struct {Disks []agentshost.DiskDiskIO []agentshost.DiskIONetwork []agentshost.NetworkInterface+ // IOCountersCollectedAt is when the cumulative DiskIO and Network counters+ // were read: stamped right after the network read, which follows the disk+ // read with only the interface listing between them.+ IOCountersCollectedAt time.Time}+// ioCountersReadNow stamps IOCountersCollectedAt; tests replace it.+var ioCountersReadNow = func() time.Time { return time.Now().UTC() }+// Collector owns the CPU baseline for one reporting loop. Its zero value is// ready to use. Retain it across reports; do not copy it after first use.type Collector struct {@@ -236,7 +244,7 @@ func (c *Collector) CollectWithDiskFilters(ctx context.Context, diskExclude, dis// ZFS ARC memory is reclaimable under pressure but is counted as "used" by// both FreeBSD (wired memory) and Linux (not in MemAvailable, openzfs/zfs#10255).// Subtract it from Used to reflect actual memory pressure. Refs: #1264/#1051- if arcSize, err := readARCSize(); err == nil && arcSize > 0 {+ if arcSize, err := readZFSARCSize(); err == nil && arcSize > 0 {if arcSize < usedBytes {usedBytes -= arcSize} else {@@ -271,6 +279,7 @@ func (c *Collector) CollectWithDiskFilters(ctx context.Context, diskExclude, dissnapshot.Disks = collectDisksWithIncludes(collectCtx, diskExclude, diskInclude)snapshot.DiskIO = collectDiskIO(collectCtx, diskExclude)snapshot.Network = collectNetwork(collectCtx)+ snapshot.IOCountersCollectedAt = ioCountersReadNow()return snapshot, nil}diff --git a/internal/hostmetrics/io_counters_collected_at_test.go b/internal/hostmetrics/io_counters_collected_at_test.gonew file mode 100644index 0000000000..438e175b90--- /dev/null+++ b/internal/hostmetrics/io_counters_collected_at_test.go@@ -0,0 +1,78 @@+package hostmetrics++import (+ "context"+ "testing"+ "time"++ gocpu "github.com/shirou/gopsutil/v4/cpu"+ godisk "github.com/shirou/gopsutil/v4/disk"+ goload "github.com/shirou/gopsutil/v4/load"+ gomem "github.com/shirou/gopsutil/v4/mem"+ gonet "github.com/shirou/gopsutil/v4/net"+)++// The counter read stamp is what the server divides counter deltas by, so it+// must mark the disk and network counter reads rather than the start of the+// snapshot: earlier steps can stall, here a memory read that takes 20s. Every+// seam the snapshot reaches is stubbed, so the test touches no host or proc+// file; it replaces package globals and must stay sequential.+func TestCollectWithDiskFiltersStampsIOCountersAtTheirRead(t *testing.T) {+ start := time.Date(2026, 10, 30, 12, 0, 0, 0, time.UTC)+ clock := start+ var diskReadAt, netReadAt time.Time++ origCounts, origPercent, origTimes, origLoad, origMemory := cpuCounts, cpuPercent, cpuTimes, loadAvg, virtualMemory+ origPartitions, origDiskIO, origInterfaces, origNetIO, origNow := diskPartitions, diskIOCounters, netInterfaces, netIOCounters, ioCountersReadNow+ origARC := readZFSARCSize+ t.Cleanup(func() {+ cpuCounts, cpuPercent, cpuTimes, loadAvg, virtualMemory = origCounts, origPercent, origTimes, origLoad, origMemory+ diskPartitions, diskIOCounters, netInterfaces, netIOCounters, ioCountersReadNow = origPartitions, origDiskIO, origInterfaces, origNetIO, origNow+ readZFSARCSize = origARC+ })++ readZFSARCSize = func() (uint64, error) { return 0, nil }+ cpuCounts = func(context.Context, bool) (int, error) { return 4, nil }+ cpuPercent = func(context.Context, time.Duration, bool) ([]float64, error) { return []float64{5}, nil }+ cpuTimes = func(context.Context, bool) ([]gocpu.TimesStat, error) {+ return []gocpu.TimesStat{{User: 10, Idle: 90}}, nil+ }+ loadAvg = func(context.Context) (*goload.AvgStat, error) { return &goload.AvgStat{}, nil }+ virtualMemory = func(context.Context) (*gomem.VirtualMemoryStat, error) {+ clock = clock.Add(20 * time.Second)+ return &gomem.VirtualMemoryStat{Total: 8 << 30, Available: 4 << 30, Free: 2 << 30, Used: 4 << 30}, nil+ }+ diskPartitions = func(context.Context, bool) ([]godisk.PartitionStat, error) { return nil, nil }+ diskIOCounters = func(context.Context, ...string) (map[string]godisk.IOCountersStat, error) {+ diskReadAt = clock+ clock = clock.Add(time.Millisecond)+ return map[string]godisk.IOCountersStat{"sda": {Name: "sda", ReadBytes: 1 << 20, WriteBytes: 2 << 20}}, nil+ }+ netInterfaces = func(context.Context) (gonet.InterfaceStatList, error) {+ clock = clock.Add(time.Millisecond)+ return gonet.InterfaceStatList{{Name: "eth0", Addrs: gonet.InterfaceAddrList{{Addr: "192.0.2.10/24"}}}}, nil+ }+ netIOCounters = func(context.Context, bool) ([]gonet.IOCountersStat, error) {+ netReadAt = clock+ return []gonet.IOCountersStat{{Name: "eth0", BytesRecv: 3 << 20, BytesSent: 4 << 20}}, nil+ }+ ioCountersReadNow = func() time.Time { return clock }++ var collector Collector+ snapshot, err := collector.CollectWithDiskFilters(context.Background(), nil, nil)+ if err != nil {+ t.Fatalf("CollectWithDiskFilters: %v", err)+ }+ if len(snapshot.DiskIO) != 1 || len(snapshot.Network) != 1 {+ t.Fatalf("snapshot counters = disk %+v network %+v, want one of each", snapshot.DiskIO, snapshot.Network)+ }+ if !snapshot.IOCountersCollectedAt.Equal(netReadAt) {+ t.Fatalf("IOCountersCollectedAt = %v, want the network counter read at %v", snapshot.IOCountersCollectedAt, netReadAt)+ }+ if lag := snapshot.IOCountersCollectedAt.Sub(diskReadAt); lag < 0 || lag > 2*time.Millisecond {+ t.Fatalf("stamp is %v after the disk counter read, want only the interface listing between them", lag)+ }+ if snapshot.IOCountersCollectedAt.Sub(start) < 20*time.Second {+ t.Fatalf("stamp %v predates the stalled memory read; it must mark the counter reads", snapshot.IOCountersCollectedAt)+ }+}diff --git a/internal/monitoring/agent_report_counter_clock_test.go b/internal/monitoring/agent_report_counter_clock_test.gonew file mode 100644index 0000000000..f8609cf74c--- /dev/null+++ b/internal/monitoring/agent_report_counter_clock_test.go@@ -0,0 +1,502 @@+package monitoring++import (+ "fmt"+ "math"+ "reflect"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"+)++// counterClockAgent is a fake Unified Agent. It collects once per interval on+// its own clock and its cumulative counters grow at constant rates, so every+// pair of its samples has one true rate. Pulse receipt time is whatever the+// test's back-to-back calls take: the reconnect burst of a drained FIFO.+type counterClockAgent struct {+ start time.Time+ interval time.Duration+}++const (+ counterClockNetInPerSecond = 1_000_000+ counterClockNetOutPerSecond = 250_000+ counterClockReadPerSecond = 4_000_000+ counterClockWritePerSecond = 2_000_000+ // Busy milliseconds per second: ten per cent of the interval.+ counterClockBusyMsPerSecond = 100+)++var counterClockTrueRates = map[string]float64{+ "agent netin": counterClockNetInPerSecond,+ "agent netout": counterClockNetOutPerSecond,+ "agent diskread": counterClockReadPerSecond,+ "agent diskwrite": counterClockWritePerSecond,+ "sda diskread": counterClockReadPerSecond,+ "sda diskwrite": counterClockWritePerSecond,+ "sda disk": counterClockBusyMsPerSecond / 10.0,+}++func newCounterClockAgent(start time.Time) counterClockAgent {+ return counterClockAgent{start: start.UTC().Truncate(time.Second), interval: 30 * time.Second}+}++func (a counterClockAgent) collectedAt(slot int) time.Time {+ return a.start.Add(time.Duration(slot) * a.interval)+}++func (a counterClockAgent) counter(perSecond uint64, slot int) uint64 {+ return 1_000_000_000_000 + perSecond*uint64(slot)*uint64(a.interval/time.Second)+}++// hostReport is the report holding slot's counters, authored at authoredAt+// and sequenced in stream. An empty stream is a legacy report, and a zero+// authoredAt a report without a timestamp. Host.UptimeSeconds marks the slot.+func (a counterClockAgent) hostReport(stream string, seq uint64, slot int, authoredAt time.Time) agentshost.Report {+ return agentshost.Report{+ Agent: agentshost.AgentInfo{ID: "counter-clock-agent", Version: "6.6.0", Type: "unified", IntervalSeconds: int(a.interval / time.Second)},+ Host: agentshost.HostInfo{+ ID: "counter-clock-host",+ MachineID: "counter-clock-machine",+ Hostname: "counter-clock-host",+ Platform: "linux",+ UptimeSeconds: int64(slot),+ },+ Network: []agentshost.NetworkInterface{{+ Name: "eth0",+ RXBytes: a.counter(counterClockNetInPerSecond, slot),+ TXBytes: a.counter(counterClockNetOutPerSecond, slot),+ }},+ DiskIO: []agentshost.DiskIO{{+ Device: "sda",+ ReadBytes: a.counter(counterClockReadPerSecond, slot),+ WriteBytes: a.counter(counterClockWritePerSecond, slot),+ IOTime: a.counter(counterClockBusyMsPerSecond, slot),+ }},+ Timestamp: authoredAt,+ SequenceID: agentshost.FormatReportSequenceID(stream, seq),+ }+}++func newCounterClockHostMonitor(t *testing.T) *Monitor {+ t.Helper()+ previous := mock.IsMockEnabled()+ mustSetMockEnabled(t, false)+ t.Cleanup(func() { mustSetMockEnabled(t, previous) })+ return &Monitor{+ state: models.NewState(),+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil)),+ hostContinuityStore: config.NewHostContinuityStore(t.TempDir(), nil),+ rateTracker: NewRateTracker(),+ metricsHistory: NewMetricsHistory(1000, 24*time.Hour),+ }+}++// applyCounterClockHost applies report and requires it to own host state.+func applyCounterClockHost(t *testing.T, m *Monitor, report agentshost.Report) models.Host {+ t.Helper()+ host, err := m.ApplyHostReport(report, nil)+ if err != nil {+ t.Fatalf("ApplyHostReport slot %d: %v", report.Host.UptimeSeconds, err)+ }+ if host.UptimeSeconds != report.Host.UptimeSeconds {+ t.Fatalf("slot %d report %q was not applied; host holds slot %d", report.Host.UptimeSeconds, report.SequenceID, host.UptimeSeconds)+ }+ return host+}++func wantTrueRate(t *testing.T, what string, got, want float64) {+ t.Helper()+ if math.Abs(got-want) > want*1e-9 {+ t.Fatalf("%s = %.1f, want the agent's real rate %.1f", what, got, want)+ }+}++func wantHostTrueRates(t *testing.T, label string, host models.Host) {+ t.Helper()+ wantTrueRate(t, label+" NetInRate", host.NetInRate, counterClockNetInPerSecond)+ wantTrueRate(t, label+" NetOutRate", host.NetOutRate, counterClockNetOutPerSecond)+ wantTrueRate(t, label+" DiskReadRate", host.DiskReadRate, counterClockReadPerSecond)+ wantTrueRate(t, label+" DiskWriteRate", host.DiskWriteRate, counterClockWritePerSecond)+}++// wantHostRatesUnknown requires host to carry no rate. Every fake counter+// moves, so a measured rate would be positive.+func wantHostRatesUnknown(t *testing.T, label string, host models.Host) {+ t.Helper()+ if host.NetInRate != 0 || host.NetOutRate != 0 || host.DiskReadRate != 0 || host.DiskWriteRate != 0 {+ t.Fatalf("%s reported rates without a comparable interval: in %.1f out %.1f read %.1f write %.1f",+ label, host.NetInRate, host.NetOutRate, host.DiskReadRate, host.DiskWriteRate)+ }+}++// counterClockHistory returns every rate point recorded for the host and its+// sda disk, by series.+func counterClockHistory(m *Monitor, hostID string) map[string][]float64 {+ series := make(map[string][]float64)+ for _, metric := range []string{"netin", "netout", "diskread", "diskwrite"} {+ for _, point := range m.metricsHistory.GetGuestMetrics("agent:"+hostID, metric, 24*time.Hour) {+ series["agent "+metric] = append(series["agent "+metric], point.Value)+ }+ }+ diskID := hostID + ":" + normalizeHostDiskDevice("sda")+ for _, metric := range []string{"diskread", "diskwrite", "disk"} {+ for _, point := range m.metricsHistory.GetDiskMetrics(diskID, metric, 24*time.Hour) {+ series["sda "+metric] = append(series["sda "+metric], point.Value)+ }+ }+ return series+}++// wantHistoryTrueRates requires every rate series to hold points and every+// point to be the agent's real rate.+func wantHistoryTrueRates(t *testing.T, m *Monitor, hostID string) {+ t.Helper()+ series := counterClockHistory(m, hostID)+ for name, want := range counterClockTrueRates {+ if len(series[name]) == 0 {+ t.Fatalf("%s history is empty", name)+ }+ for i, got := range series[name] {+ wantTrueRate(t, fmt.Sprintf("%s history point %d", name, i), got, want)+ }+ }+}++// A FIFO drained on reconnect delivers reports collected 30s apart within+// milliseconds of each other. Each is newer than the last accepted report, so+// the stream watermark admits them all, and each must measure its counters+// over the 30s it really covers rather than over the receipt gap.+func TestApplyHostReportDrainedFIFORatesUseAuthoredCollectionInterval(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ // Collected two days ago and held through the outage, so rate-series+ // cleanup must judge freshness by receipt, not by the sample clock.+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))++ first := applyCounterClockHost(t, m, agent.hostReport("stream-a", 1, 1, agent.collectedAt(1)))+ wantHostRatesUnknown(t, "first sample", first)+ for slot := 2; slot <= 11; slot++ {+ host := applyCounterClockHost(t, m, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)))+ wantHostTrueRates(t, fmt.Sprintf("drained slot %d", slot), host)+ }+ wantHistoryTrueRates(t, m, first.ID)++ // Every series was received just now. Expiring it by its two-day-old+ // sample time would leave the next report without a baseline.+ m.cleanupRateTracker(time.Now())+ next := applyCounterClockHost(t, m, agent.hostReport("stream-a", 12, 12, agent.collectedAt(12)))+ wantHostTrueRates(t, "report after rate-series cleanup", next)+}++// An agent restart opens a new stream on the same machine clock. Its first+// report is measured from the old stream's last accepted report, and a+// delayed old-stream report is rejected without touching rates or history.+func TestApplyHostReportStreamChangeKeepsMachineClockAndRetiresOldStream(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ for slot := 1; slot <= 3; slot++ {+ applyCounterClockHost(t, m, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)))+ }++ restarted := applyCounterClockHost(t, m, agent.hostReport("stream-b", 1, 4, agent.collectedAt(4)))+ wantHostTrueRates(t, "first report of the new stream", restarted)++ before := counterClockHistory(m, restarted.ID)+ delayed, err := m.ApplyHostReport(agent.hostReport("stream-a", 4, 3, agent.collectedAt(3)), nil)+ if err != nil {+ t.Fatalf("ApplyHostReport delayed old-stream report: %v", err)+ }+ if delayed.UptimeSeconds != 4 {+ t.Fatalf("retired-stream report replaced state: host holds slot %d, want 4", delayed.UptimeSeconds)+ }+ wantHostTrueRates(t, "stored host returned for the retired stream", delayed)+ if after := counterClockHistory(m, restarted.ID); !reflect.DeepEqual(after, before) {+ t.Fatalf("retired-stream report wrote rate history:\nbefore %v\nafter %v", before, after)+ }++ next := applyCounterClockHost(t, m, agent.hostReport("stream-b", 2, 5, agent.collectedAt(5)))+ wantHostTrueRates(t, "second report of the new stream", next)+ wantHistoryTrueRates(t, m, next.ID)+}++// Each clock transition either has a comparable interval on one clock or has+// none. Without one, no rate is reported and no rate history is written:+// neither an authored time subtracted from a receipt baseline, nor a huge or+// zero rate from an interval that went backwards or did not advance.+func TestApplyHostReportClockTransitionsReportUnknownWithoutComparableInterval(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))+ steppedBack := func(slot int) time.Time { return agent.collectedAt(slot).Add(-time.Hour) }+ steppedForward := func(slot int) time.Time { return agent.collectedAt(slot).Add(time.Hour) }++ // A legacy report sets a receipt-time baseline.+ hostID := applyCounterClockHost(t, m, agent.hostReport("", 0, 1, agent.collectedAt(1))).ID++ steps := []struct {+ name string+ seq uint64+ slot int+ authoredAt time.Time+ want string+ }{+ {"upgrade to a sequenced stream after a receipt-time baseline", 1, 2, agent.collectedAt(2), "unknown"},+ {"second authored sample", 2, 3, agent.collectedAt(3), "true"},+ {"sequenced report without a timestamp", 3, 4, time.Time{}, "unknown"},+ {"timestamp returns", 4, 5, agent.collectedAt(5), "unknown"},+ {"authored interval again", 5, 6, agent.collectedAt(6), "true"},+ {"clock steps back an hour", 6, 7, steppedBack(7), "unknown"},+ {"first sample on the stepped clock", 7, 8, steppedBack(8), "unknown"},+ {"interval on the stepped clock", 8, 9, steppedBack(9), "true"},+ {"equal timestamp with the next sequence", 9, 10, steppedBack(9), "unknown"},+ {"first sample after the repeated timestamp", 10, 11, steppedBack(11), "unknown"},+ {"interval after the repeated timestamp", 11, 12, steppedBack(12), "true"},+ // A forward step is indistinguishable from a longer collection gap,+ // which rates average over: the sample may under-report, never spike.+ {"clock steps forward two hours", 12, 13, steppedForward(13), "no spike"},+ {"interval on the forward clock", 13, 14, steppedForward(14), "true"},+ }+ for _, step := range steps {+ before := counterClockHistory(m, hostID)+ host := applyCounterClockHost(t, m, agent.hostReport("stream-a", step.seq, step.slot, step.authoredAt))+ switch step.want {+ case "unknown":+ wantHostRatesUnknown(t, step.name, host)+ if after := counterClockHistory(m, hostID); !reflect.DeepEqual(after, before) {+ t.Fatalf("%s wrote rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)+ }+ case "true":+ wantHostTrueRates(t, step.name, host)+ case "no spike":+ for _, rate := range []struct {+ what string+ got, want float64+ }{+ {"NetInRate", host.NetInRate, counterClockNetInPerSecond},+ {"NetOutRate", host.NetOutRate, counterClockNetOutPerSecond},+ {"DiskReadRate", host.DiskReadRate, counterClockReadPerSecond},+ {"DiskWriteRate", host.DiskWriteRate, counterClockWritePerSecond},+ } {+ if rate.got <= 0 || rate.got > rate.want {+ t.Fatalf("%s %s = %.1f, want a positive rate no higher than the real %.1f", step.name, rate.what, rate.got, rate.want)+ }+ }+ }+ }+ for name, values := range counterClockHistory(m, hostID) {+ for i, got := range values {+ if got <= 0 || got > counterClockTrueRates[name]*(1+1e-9) {+ t.Fatalf("%s history point %d = %.1f, want a positive rate no higher than %.1f", name, i, got, counterClockTrueRates[name])+ }+ }+ }+}++// Reports delivered at the same time race for the per-host lock. Whatever+// order they win in, the watermark admits an increasing subset, each accepted+// report is measured over the authored interval since the previously accepted+// one, a rejected report returns the stored host, and the newest report owns+// state.+func TestApplyHostReportConcurrentDeliveryMeasuresAcceptedReportsOnAuthoredClock(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ first := applyCounterClockHost(t, m, agent.hostReport("stream-a", 1, 1, agent.collectedAt(1)))++ const last = 16+ results := make([]models.Host, last+1)+ errs := make([]error, last+1)+ release := make(chan struct{})+ var wg sync.WaitGroup+ for slot := 2; slot <= last; slot++ {+ wg.Add(1)+ go func(slot int) {+ defer wg.Done()+ <-release+ results[slot], errs[slot] = m.ApplyHostReport(agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot)), nil)+ }(slot)+ }+ close(release)+ wg.Wait()++ for slot := 2; slot <= last; slot++ {+ if errs[slot] != nil {+ t.Fatalf("ApplyHostReport slot %d: %v", slot, errs[slot])+ }+ wantHostTrueRates(t, fmt.Sprintf("concurrent slot %d", slot), results[slot])+ }+ stored, ok := m.state.GetHost(first.ID)+ if !ok || stored.UptimeSeconds != last {+ t.Fatalf("newest report must own state: found %v holding slot %d, want slot %d", ok, stored.UptimeSeconds, last)+ }+ wantHostTrueRates(t, "stored host", stored)+ wantHistoryTrueRates(t, m, first.ID)+}++// Legacy reports carry no stream, so their timestamps only guard the+// reconnect burst and never time a rate: rates stay on Pulse receipt time.+func TestApplyHostReportLegacyRatesStayOnReceiptTime(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ host := applyCounterClockHost(t, m, agent.hostReport("", 0, 1, agent.collectedAt(1)))++ // Unchanged counters are measured idle over the receipt interval.+ idle := agent.hostReport("", 0, 1, agent.collectedAt(2))+ idle.Host.UptimeSeconds = 2+ idleSentAt := time.Now()+ applyCounterClockHost(t, m, idle)+ points := m.metricsHistory.GetGuestMetrics("agent:"+host.ID, "netin", 24*time.Hour)+ if len(points) != 1 || points[0].Value != 0 {+ t.Fatalf("unchanged legacy counters must be measured idle: %+v", points)+ }++ // Two slots of counters authored one interval later: an authored-clock+ // rate would be twice the real rate, while the receipt interval is at+ // most the wall time these calls took.+ moved := applyCounterClockHost(t, m, agent.hostReport("", 0, 3, agent.collectedAt(3)))+ wall := time.Since(idleSentAt).Seconds()+ delta := float64(agent.counter(counterClockNetInPerSecond, 3) - agent.counter(counterClockNetInPerSecond, 1))+ if moved.NetInRate < delta/wall {+ t.Fatalf("legacy NetInRate = %.1f, want the receipt-interval rate of at least %.1f (authored interval would give %.1f)",+ moved.NetInRate, delta/wall, delta/agent.interval.Seconds())+ }++ // A backwards legacy timestamp inside the reconnect burst stays rejected.+ before := counterClockHistory(m, host.ID)+ late := agent.hostReport("", 0, 2, agent.collectedAt(2))+ stored, err := m.ApplyHostReport(late, nil)+ if err != nil {+ t.Fatalf("ApplyHostReport late legacy report: %v", err)+ }+ if stored.UptimeSeconds != 3 {+ t.Fatalf("late legacy report replaced state: host holds slot %d, want 3", stored.UptimeSeconds)+ }+ if after := counterClockHistory(m, host.ID); !reflect.DeepEqual(after, before) {+ t.Fatalf("late legacy report wrote rate history:\nbefore %v\nafter %v", before, after)+ }+}++// The Docker module follows the same stream ordering. A drained FIFO measures+// container counters on the authored interval, and a helper-degraded status+// report between complete reports neither replaces the inventory nor moves a+// rate baseline, so the next complete report spans both intervals.+func TestApplyDockerReportDrainedFIFOContainerRatesUseAuthoredCollectionInterval(t *testing.T) {+ previous := mock.IsMockEnabled()+ mustSetMockEnabled(t, false)+ t.Cleanup(func() { mustSetMockEnabled(t, previous) })+ m := newTestMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))++ report := func(seq uint64, slot int, complete bool) agentsdocker.Report {+ inventoryComplete := complete+ r := agentsdocker.Report{+ Agent: agentsdocker.AgentInfo{ID: "counter-clock-docker", Version: "6.6.0", Type: "unified", IntervalSeconds: 30},+ Host: agentsdocker.HostInfo{Hostname: "counter-clock-docker-host"},+ InventoryComplete: &inventoryComplete,+ Timestamp: agent.collectedAt(slot),+ SequenceID: agentshost.FormatReportSequenceID("docker-stream", seq),+ }+ if complete {+ r.Containers = []agentsdocker.Container{{+ ID: "counter-clock-container",+ Name: "api",+ State: "running",+ NetworkRXBytes: agent.counter(counterClockNetInPerSecond, slot),+ NetworkTXBytes: agent.counter(counterClockNetOutPerSecond, slot),+ BlockIO: &agentsdocker.ContainerBlockIO{+ ReadBytes: agent.counter(counterClockReadPerSecond, slot),+ WriteBytes: agent.counter(counterClockWritePerSecond, slot),+ },+ }}+ }+ return r+ }+ apply := func(seq uint64, slot int, complete bool) models.DockerContainer {+ t.Helper()+ host, err := m.ApplyDockerReport(report(seq, slot, complete), nil)+ if err != nil {+ t.Fatalf("ApplyDockerReport slot %d: %v", slot, err)+ }+ if len(host.Containers) != 1 {+ t.Fatalf("slot %d containers = %+v, want the one running container", slot, host.Containers)+ }+ return host.Containers[0]+ }+ wantContainerTrueRates := func(label string, container models.DockerContainer) {+ t.Helper()+ wantTrueRate(t, label+" NetInRate", container.NetInRate, counterClockNetInPerSecond)+ wantTrueRate(t, label+" NetOutRate", container.NetOutRate, counterClockNetOutPerSecond)+ if container.BlockIO == nil || container.BlockIO.ReadRateBytesPerSecond == nil || container.BlockIO.WriteRateBytesPerSecond == nil {+ t.Fatalf("%s block I/O rates missing: %+v", label, container.BlockIO)+ }+ wantTrueRate(t, label+" read rate", *container.BlockIO.ReadRateBytesPerSecond, counterClockReadPerSecond)+ wantTrueRate(t, label+" write rate", *container.BlockIO.WriteRateBytesPerSecond, counterClockWritePerSecond)+ }+ containerHistory := func() map[string][]float64 {+ series := make(map[string][]float64)+ for _, metric := range []string{"netin", "netout", "diskread", "diskwrite"} {+ for _, point := range m.metricsHistory.GetGuestMetrics("docker:counter-clock-container", metric, 24*time.Hour) {+ series[metric] = append(series[metric], point.Value)+ }+ }+ return series+ }++ apply(1, 1, true)+ for metric, values := range containerHistory() {+ if len(values) != 0 {+ t.Fatalf("first sample wrote %s history without an interval: %v", metric, values)+ }+ }+ for slot := 2; slot <= 6; slot++ {+ wantContainerTrueRates(fmt.Sprintf("drained slot %d", slot), apply(uint64(slot), slot, true))+ }++ before := containerHistory()+ retained := apply(7, 7, false)+ wantContainerTrueRates("container retained through the status-only report", retained)+ if after := containerHistory(); !reflect.DeepEqual(after, before) {+ t.Fatalf("status-only report wrote container history:\nbefore %v\nafter %v", before, after)+ }++ wantContainerTrueRates("complete report after the status-only report", apply(8, 8, true))+ before = containerHistory()+ fault := report(9, 9, true)+ fault.Timestamp = agent.collectedAt(8)+ if _, err := m.ApplyDockerReport(fault, nil); err != nil {+ t.Fatal(err)+ }+ if after := containerHistory(); !reflect.DeepEqual(after, before) {+ t.Fatalf("repeated source time wrote unknown rates as history: before %v after %v", before, after)+ }+ apply(10, 10, true) // First baseline after dropping the ambiguous sample.+ wantContainerTrueRates("interval after the clock fault", apply(11, 11, true))+ series := containerHistory()+ for metric, want := range map[string]float64{+ "netin": counterClockNetInPerSecond, "netout": counterClockNetOutPerSecond,+ "diskread": counterClockReadPerSecond, "diskwrite": counterClockWritePerSecond,+ } {+ values := series[metric]+ if len(values) == 0 {+ t.Fatalf("container %s history is empty", metric)+ }+ for i, got := range values {+ wantTrueRate(t, fmt.Sprintf("container %s history point %d", metric, i), got, want)+ }+ }+ idle := report(12, 12, true)+ idle.Containers = report(11, 11, true).Containers+ if _, err := m.ApplyDockerReport(idle, nil); err != nil {+ t.Fatal(err)+ }+ for metric, values := range containerHistory() {+ if len(values) == 0 || values[len(values)-1] != 0 {+ t.Fatalf("measured idle %s did not retain zero: %v", metric, values)+ }+ }+}diff --git a/internal/monitoring/agent_report_io_counter_clock_test.go b/internal/monitoring/agent_report_io_counter_clock_test.gonew file mode 100644index 0000000000..004fb0b906--- /dev/null+++ b/internal/monitoring/agent_report_io_counter_clock_test.go@@ -0,0 +1,170 @@+package monitoring++import (+ "encoding/json"+ "fmt"+ "reflect"+ "testing"+ "time"++ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"+)++// ioCounterStamp is the JSON value of a counter read stamp.+func ioCounterStamp(t *testing.T, at time.Time) string {+ t.Helper()+ raw, err := json.Marshal(at.UTC())+ if err != nil {+ t.Fatalf("encode counter read stamp: %v", err)+ }+ return string(raw)+}++// ioCounterWireReport decodes report from the JSON a current agent sends,+// with stamp spliced in as its ioCountersCollectedAt value; an empty stamp is+// an agent that sends none. A server without the field ignores it, so these+// tests compile there and fail on its behaviour.+func ioCounterWireReport(t *testing.T, report agentshost.Report, stamp string) agentshost.Report {+ t.Helper()+ raw, err := json.Marshal(report)+ if err != nil {+ t.Fatalf("encode report: %v", err)+ }+ if stamp != "" {+ raw = append(raw[:len(raw)-1], []byte(`,"ioCountersCollectedAt":`+stamp+`}`)...)+ }+ var decoded agentshost.Report+ if err := json.Unmarshal(raw, &decoded); err != nil {+ t.Fatalf("decode wire report: %v", err)+ }+ return decoded+}++// slowTail is how long optional collectors ran after the counter read before+// the report Timestamp: 20s on even slots, none on odd ones.+func slowTail(slot int) time.Duration {+ if slot%2 == 0 {+ return 20 * time.Second+ }+ return 0+}++// Counters are read every 30s at 0/30/60/...; optional collectors then delay+// the report Timestamp to 0/50/60/110/... A drained FIFO delivers the reports+// back to back. Rates must divide by the 30s between counter reads, not by+// the 50s/10s between Timestamps or by the receipt gap.+func TestApplyHostReportSlowOptionalTailRatesUseIOCounterReadTime(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))+ report := func(slot int) agentshost.Report {+ authored := agent.collectedAt(slot).Add(slowTail(slot))+ return ioCounterWireReport(t, agent.hostReport("stream-a", uint64(slot), slot, authored), ioCounterStamp(t, agent.collectedAt(slot)))+ }++ first := applyCounterClockHost(t, m, report(1))+ wantHostRatesUnknown(t, "first sample", first)+ for slot := 2; slot <= 11; slot++ {+ host := applyCounterClockHost(t, m, report(slot))+ wantHostTrueRates(t, fmt.Sprintf("drained slot %d", slot), host)+ }+ wantHistoryTrueRates(t, m, first.ID)++ // Received just now from two-day-old reads: cleanup keyed on the sample+ // clock would drop the baselines.+ m.cleanupRateTracker(time.Now())+ wantHostTrueRates(t, "report after rate-series cleanup", applyCounterClockHost(t, m, report(12)))+}++// Negative control: the same slow-tail reports from an agent that sends no+// counter read stamp keep the report-Timestamp fallback, so their rates carry+// the tail (30/50 and 30/10 of the real rate). This proves the tail above+// reaches the denominator, and that an older agent is left as it was.+func TestApplyHostReportOlderAgentSlowTailKeepsTimestampFallback(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ report := func(slot int) agentshost.Report {+ return ioCounterWireReport(t, agent.hostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot).Add(slowTail(slot))), "")+ }+ applyCounterClockHost(t, m, report(1))+ slow := applyCounterClockHost(t, m, report(2))+ wantTrueRate(t, "older agent after a slow tail", slow.NetInRate, counterClockNetInPerSecond*30.0/50)+ fast := applyCounterClockHost(t, m, report(3))+ wantTrueRate(t, "older agent after the tail ends", fast.NetInRate, counterClockNetInPerSecond*30.0/10)+}++// Every sampling-mode transition and source clock fault either has a+// comparable interval in one mode or reports no rate and writes no rate+// history: report-Timestamp and counter-read times share a clock but not an+// offset, so a switch is a new baseline, and a stamp that does not advance+// drops it. A counter reset keeps its existing rebase to zero.+func TestApplyHostReportIOCounterReadTimeTransitions(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))+ at := agent.collectedAt+ back := func(slot int) time.Time { return at(slot).Add(-time.Hour) }+ stamp := func(readAt time.Time) string { return ioCounterStamp(t, readAt) }++ // A legacy report sets receipt-time baselines; its stamp has no ordered+ // clock behind it.+ legacy := ioCounterWireReport(t, agent.hostReport("", 0, 1, at(1)), stamp(at(1)))+ hostID := applyCounterClockHost(t, m, legacy).ID++ steps := []struct {+ name string+ stream string+ seq uint64+ slot int+ stamp string+ want string+ }{+ {"older sequenced agent after receipt-time baselines", "stream-a", 1, 2, "", "unknown"},+ {"older agent on its report Timestamp", "stream-a", 2, 3, "", "true"},+ {"agent starts stamping its counter read", "stream-a", 3, 4, stamp(at(4)), "unknown"},+ {"counter-read interval", "stream-a", 4, 5, stamp(at(5)), "true"},+ {"stamp missing", "stream-a", 5, 6, "", "unknown"},+ {"stamp returns", "stream-a", 6, 7, stamp(at(7)), "unknown"},+ {"counter-read interval again", "stream-a", 7, 8, stamp(at(8)), "true"},+ {"zero stamp", "stream-a", 8, 9, `"0001-01-01T00:00:00Z"`, "unknown"},+ {"stamp after the zero one", "stream-a", 9, 10, stamp(at(10)), "unknown"},+ {"interval after the zero stamp", "stream-a", 10, 11, stamp(at(11)), "true"},+ {"agent restart keeps the machine clock", "stream-b", 1, 12, stamp(at(12)), "true"},+ {"stamp steps back an hour", "stream-b", 2, 13, stamp(back(13)), "unknown"},+ {"first read on the stepped clock", "stream-b", 3, 14, stamp(back(14)), "unknown"},+ {"interval on the stepped clock", "stream-b", 4, 15, stamp(back(15)), "true"},+ {"equal stamp with the next sequence", "stream-b", 5, 16, stamp(back(15)), "unknown"},+ {"first read after the repeated stamp", "stream-b", 6, 17, stamp(back(17)), "unknown"},+ {"interval after the repeated stamp", "stream-b", 7, 18, stamp(back(18)), "true"},+ }+ for _, step := range steps {+ before := counterClockHistory(m, hostID)+ // Older-agent steps carry no tail so their Timestamp interval is exact.+ report := ioCounterWireReport(t, agent.hostReport(step.stream, step.seq, step.slot, at(step.slot)), step.stamp)+ host := applyCounterClockHost(t, m, report)+ switch step.want {+ case "unknown":+ wantHostRatesUnknown(t, step.name, host)+ if after := counterClockHistory(m, hostID); !reflect.DeepEqual(after, before) {+ t.Fatalf("%s wrote rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)+ }+ case "true":+ wantHostTrueRates(t, step.name, host)+ }+ }++ // The host reboots: every counter restarts from zero. That keeps its+ // existing meaning, a measured zero that rebases, and the next read is+ // measured from the restarted counters.+ rebooted := agent.hostReport("stream-b", 8, 19, back(19))+ rebooted.Network[0].RXBytes, rebooted.Network[0].TXBytes = 0, 0+ rebooted.DiskIO[0].ReadBytes, rebooted.DiskIO[0].WriteBytes, rebooted.DiskIO[0].IOTime = 0, 0, 0+ host := applyCounterClockHost(t, m, ioCounterWireReport(t, rebooted, stamp(back(19))))+ points := m.metricsHistory.GetGuestMetrics("agent:"+hostID, "netin", 24*time.Hour)+ if host.NetInRate != 0 || len(points) == 0 || points[len(points)-1].Value != 0 {+ t.Fatalf("counter reset = rate %.1f, last history %+v, want the existing measured-zero rebase", host.NetInRate, points)+ }+ next := agent.hostReport("stream-b", 9, 20, back(20))+ slots := uint64(agent.interval / time.Second)+ next.Network[0].RXBytes, next.Network[0].TXBytes = counterClockNetInPerSecond*slots, counterClockNetOutPerSecond*slots+ next.DiskIO[0].ReadBytes, next.DiskIO[0].WriteBytes, next.DiskIO[0].IOTime = counterClockReadPerSecond*slots, counterClockWritePerSecond*slots, counterClockBusyMsPerSecond*slots+ wantHostTrueRates(t, "first interval after the reset", applyCounterClockHost(t, m, ioCounterWireReport(t, next, stamp(back(20)))))+}diff --git a/internal/monitoring/agent_report_libvirt_clock_test.go b/internal/monitoring/agent_report_libvirt_clock_test.gonew file mode 100644index 0000000000..5bcf3c2590--- /dev/null+++ b/internal/monitoring/agent_report_libvirt_clock_test.go@@ -0,0 +1,326 @@+package monitoring++import (+ "fmt"+ "math"+ "reflect"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"+)++const (+ // One vCPU-second per second across four vCPUs: 25% of capacity.+ libvirtClockCPUNanosPerSecond = 1_000_000_000+ libvirtClockVCPUs = 4+ libvirtClockNetInPerSecond = 500_000+ libvirtClockNetOutPerSecond = 125_000+ libvirtClockReadPerSecond = 2_000_000+ libvirtClockWritePerSecond = 1_000_000+ // The agent stamps CollectedAt before it reads domain stats and the report+ // Timestamp after every collector, so CollectedAt sits earlier.+ libvirtClockCollectedLead = 7 * time.Second+)++var libvirtClockTrueRates = map[string]float64{+ "cpu": 100.0 / libvirtClockVCPUs,+ "netin": libvirtClockNetInPerSecond,+ "netout": libvirtClockNetOutPerSecond,+ "diskread": libvirtClockReadPerSecond,+ "diskwrite": libvirtClockWritePerSecond,+}++func TestApplyHostReportLibvirtCPUCounterRetainsUnsignedRange(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ for slot := 1; slot <= 3; slot++ {+ domain := agent.libvirtDomain(slot, 0)+ // A busy, long-running domain can accumulate more than MaxInt64+ // nanoseconds across its vCPUs. Clamping both samples loses the delta.+ domain.CPUTimeNanoseconds = uint64(math.MaxInt64) - 15_000_000_000 + uint64(slot)*30_000_000_000+ report := agent.libvirtHostReport("stream-a", uint64(slot), slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), domain)+ got := libvirtClockDomain(t, "unsigned CPU counter", applyCounterClockHost(t, m, report))+ if slot > 1 {+ wantLibvirtTrueRates(t, "unsigned CPU counter", got)+ }+ }+}++// libvirtCollectedAt is when the fake agent read slot's domain stats.+func (a counterClockAgent) libvirtCollectedAt(slot int) time.Time {+ return a.collectedAt(slot).Add(-libvirtClockCollectedLead)+}++// libvirtCounter is a domain counter at slot. A domain restarted at+// restartSlot counts again from zero; restartSlot 0 means it never restarted.+func (a counterClockAgent) libvirtCounter(perSecond uint64, slot, restartSlot int) uint64 {+ if restartSlot == 0 {+ return a.counter(perSecond, slot)+ }+ return perSecond * uint64(slot-restartSlot) * uint64(a.interval/time.Second)+}++func (a counterClockAgent) libvirtDomain(slot, restartSlot int) agentshost.LibvirtDomain {+ return agentshost.LibvirtDomain{+ Name: "app",+ State: "running",+ VCPUs: libvirtClockVCPUs,+ CPUTimeNanoseconds: a.libvirtCounter(libvirtClockCPUNanosPerSecond, slot, restartSlot),+ MemoryCurrentBytes: 2 << 30,+ MemoryMaximumBytes: 4 << 30,+ NetworkRXBytes: a.libvirtCounter(libvirtClockNetInPerSecond, slot, restartSlot),+ NetworkTXBytes: a.libvirtCounter(libvirtClockNetOutPerSecond, slot, restartSlot),+ DiskReadBytes: a.libvirtCounter(libvirtClockReadPerSecond, slot, restartSlot),+ DiskWriteBytes: a.libvirtCounter(libvirtClockWritePerSecond, slot, restartSlot),+ }+}++// libvirtHostReport is hostReport carrying a readable libvirt inventory+// collected at collectedAt (zero: an inventory without CollectedAt), holding+// the "app" domain unless it is omitted.+func (a counterClockAgent) libvirtHostReport(stream string, seq uint64, slot int, authoredAt, collectedAt time.Time, domains ...agentshost.LibvirtDomain) agentshost.Report {+ report := a.hostReport(stream, seq, slot, authoredAt)+ report.Libvirt = &agentshost.LibvirtInventory{+ Domains: append([]agentshost.LibvirtDomain{}, domains...),+ CollectedAt: collectedAt,+ }+ return report+}++func libvirtClockDomain(t *testing.T, label string, host models.Host) models.HostLibvirtDomain {+ t.Helper()+ if host.Libvirt == nil || len(host.Libvirt.Domains) != 1 {+ t.Fatalf("%s libvirt inventory = %+v, want the app domain", label, host.Libvirt)+ }+ return host.Libvirt.Domains[0]+}++func wantLibvirtTrueRates(t *testing.T, label string, domain models.HostLibvirtDomain) {+ t.Helper()+ if !domain.CPUUsageValid || !domain.IORatesValid {+ t.Fatalf("%s reported no rate (cpu valid %v, io valid %v), want the domain's real rates", label, domain.CPUUsageValid, domain.IORatesValid)+ }+ wantTrueRate(t, label+" CPU usage", domain.CPUUsagePercent, libvirtClockTrueRates["cpu"])+ wantTrueRate(t, label+" NetworkInRate", domain.NetworkInRate, libvirtClockNetInPerSecond)+ wantTrueRate(t, label+" NetworkOutRate", domain.NetworkOutRate, libvirtClockNetOutPerSecond)+ wantTrueRate(t, label+" DiskReadRate", domain.DiskReadRate, libvirtClockReadPerSecond)+ wantTrueRate(t, label+" DiskWriteRate", domain.DiskWriteRate, libvirtClockWritePerSecond)+}++func wantLibvirtUnknown(t *testing.T, label string, domain models.HostLibvirtDomain) {+ t.Helper()+ if domain.CPUUsageValid || domain.IORatesValid {+ t.Fatalf("%s reported rates without a comparable interval: cpu %.1f (valid %v), in %.1f (valid %v)",+ label, domain.CPUUsagePercent, domain.CPUUsageValid, domain.NetworkInRate, domain.IORatesValid)+ }+}++// libvirtClockHistory returns every CPU and I/O history point recorded for+// the app domain, by metric.+func libvirtClockHistory(m *Monitor, hostID string) map[string][]float64 {+ metricID := libvirtDomainMetricID(hostID, agentLibvirtDomainID("app"))+ series := make(map[string][]float64)+ for metric := range libvirtClockTrueRates {+ for _, point := range m.metricsHistory.GetGuestMetrics(metricID, metric, 24*time.Hour) {+ series[metric] = append(series[metric], point.Value)+ }+ }+ return series+}++func wantLibvirtHistoryTrueRates(t *testing.T, m *Monitor, hostID string) {+ t.Helper()+ series := libvirtClockHistory(m, hostID)+ for metric, want := range libvirtClockTrueRates {+ if len(series[metric]) == 0 {+ t.Fatalf("libvirt %s history is empty", metric)+ }+ for i, got := range series[metric] {+ wantTrueRate(t, fmt.Sprintf("libvirt %s history point %d", metric, i), got, want)+ }+ }+}++// A drained FIFO delivers host reports collected 30s apart within+// milliseconds. The libvirt domain's CPU and I/O counters inside each report+// must be measured over the 30s between their CollectedAt stamps, not over the+// receipt gap, and not over the report Timestamp, which the agent stamps after+// a variable-length tail of other collectors.+func TestApplyHostReportDrainedFIFOLibvirtRatesUseCollectedAt(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-48 * time.Hour))+ authoredAt := func(slot int) time.Time {+ return agent.collectedAt(slot).Add(time.Duration(slot%3) * 4 * time.Second)+ }+ report := func(slot int) agentshost.Report {+ return agent.libvirtHostReport("stream-a", uint64(slot), slot, authoredAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))+ }++ first := applyCounterClockHost(t, m, report(1))+ wantLibvirtUnknown(t, "first sample", libvirtClockDomain(t, "first sample", first))+ for slot := 2; slot <= 11; slot++ {+ host := applyCounterClockHost(t, m, report(slot))+ wantLibvirtTrueRates(t, fmt.Sprintf("drained slot %d", slot), libvirtClockDomain(t, "drained", host))+ }+ wantLibvirtHistoryTrueRates(t, m, first.ID)++ // Both domain series were received just now; cleanup keyed on their+ // two-day-old sample clock would drop the baselines.+ m.cleanupRateTracker(time.Now())+ next := applyCounterClockHost(t, m, report(12))+ wantLibvirtTrueRates(t, "report after rate-series cleanup", libvirtClockDomain(t, "after cleanup", next))+}++// Each libvirt clock transition either has a comparable CollectedAt interval+// or reports no CPU or I/O rate and writes no rate history for that sample.+func TestApplyHostReportLibvirtClockTransitionsReportUnknownWithoutComparableInterval(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-6 * time.Hour))+ steppedBack := func(slot int) time.Time { return agent.libvirtCollectedAt(slot).Add(-time.Hour) }++ // A legacy report sets receipt-time baselines even though it carries+ // CollectedAt: without a stream it has no ordered source clock.+ legacy := agent.libvirtHostReport("", 0, 1, agent.collectedAt(1), agent.libvirtCollectedAt(1), agent.libvirtDomain(1, 0))+ hostID := applyCounterClockHost(t, m, legacy).ID++ steps := []struct {+ name string+ seq uint64+ slot int+ authoredAt time.Time+ collectedAt time.Time+ want string+ }{+ {"upgrade to a sequenced stream after receipt-time baselines", 1, 2, agent.collectedAt(2), agent.libvirtCollectedAt(2), "unknown"},+ {"second CollectedAt sample", 2, 3, agent.collectedAt(3), agent.libvirtCollectedAt(3), "true"},+ {"report without a Timestamp keeps the inventory's own clock", 3, 4, time.Time{}, agent.libvirtCollectedAt(4), "true"},+ {"inventory without CollectedAt", 4, 5, agent.collectedAt(5), time.Time{}, "unknown"},+ {"CollectedAt returns", 5, 6, agent.collectedAt(6), agent.libvirtCollectedAt(6), "unknown"},+ {"CollectedAt interval again", 6, 7, agent.collectedAt(7), agent.libvirtCollectedAt(7), "true"},+ {"clock steps back an hour", 7, 8, agent.collectedAt(8).Add(-time.Hour), steppedBack(8), "unknown"},+ {"first sample on the stepped clock", 8, 9, agent.collectedAt(9).Add(-time.Hour), steppedBack(9), "unknown"},+ {"interval on the stepped clock", 9, 10, agent.collectedAt(10).Add(-time.Hour), steppedBack(10), "true"},+ {"equal CollectedAt with the next sequence", 10, 11, agent.collectedAt(11).Add(-time.Hour), steppedBack(10), "unknown"},+ {"first sample after the repeated CollectedAt", 11, 12, agent.collectedAt(12).Add(-time.Hour), steppedBack(12), "unknown"},+ {"interval after the repeated CollectedAt", 12, 13, agent.collectedAt(13).Add(-time.Hour), steppedBack(13), "true"},+ }+ for _, step := range steps {+ before := libvirtClockHistory(m, hostID)+ host := applyCounterClockHost(t, m, agent.libvirtHostReport("stream-a", step.seq, step.slot, step.authoredAt, step.collectedAt, agent.libvirtDomain(step.slot, 0)))+ domain := libvirtClockDomain(t, step.name, host)+ switch step.want {+ case "unknown":+ wantLibvirtUnknown(t, step.name, domain)+ if after := libvirtClockHistory(m, hostID); !reflect.DeepEqual(after, before) {+ t.Fatalf("%s wrote libvirt rate history without a comparable interval:\nbefore %v\nafter %v", step.name, before, after)+ }+ case "true":+ wantLibvirtTrueRates(t, step.name, domain)+ }+ }+ wantLibvirtHistoryTrueRates(t, m, hostID)+}++// An agent restart keeps the machine clock, so the new stream's first libvirt+// sample is measured from the old stream's last, and a delayed old-stream+// report changes neither the domain nor its history.+func TestApplyHostReportLibvirtStreamChangeKeepsMachineClock(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ report := func(stream string, seq uint64, slot int) agentshost.Report {+ return agent.libvirtHostReport(stream, seq, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))+ }+ for slot := 1; slot <= 3; slot++ {+ applyCounterClockHost(t, m, report("stream-a", uint64(slot), slot))+ }++ restarted := applyCounterClockHost(t, m, report("stream-b", 1, 4))+ wantLibvirtTrueRates(t, "first report of the new stream", libvirtClockDomain(t, "restarted", restarted))++ before := libvirtClockHistory(m, restarted.ID)+ delayed, err := m.ApplyHostReport(report("stream-a", 4, 3), nil)+ if err != nil {+ t.Fatalf("ApplyHostReport delayed old-stream report: %v", err)+ }+ if delayed.UptimeSeconds != 4 {+ t.Fatalf("retired-stream report replaced state: host holds slot %d, want 4", delayed.UptimeSeconds)+ }+ wantLibvirtTrueRates(t, "stored domain returned for the retired stream", libvirtClockDomain(t, "delayed", delayed))+ if after := libvirtClockHistory(m, restarted.ID); !reflect.DeepEqual(after, before) {+ t.Fatalf("retired-stream report wrote libvirt history:\nbefore %v\nafter %v", before, after)+ }++ next := applyCounterClockHost(t, m, report("stream-b", 2, 5))+ wantLibvirtTrueRates(t, "second report of the new stream", libvirtClockDomain(t, "next", next))+}++// Omissions keep their existing meaning: a readable inventory without the+// domain retires it, so CPU usage restarts on its return, while its+// cumulative I/O counters measure the whole span since their last sample; an+// inventory the agent failed to read preserves the last one, and the next+// read measures CPU from that last read rather than from the host report in+// between; a domain whose counters restart has no CPU usage for that sample.+func TestApplyHostReportLibvirtOmittedDomainInventoryAndRestart(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ report := func(seq uint64, slot int, domains ...agentshost.LibvirtDomain) agentshost.Report {+ return agent.libvirtHostReport("stream-a", seq, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), domains...)+ }+ for slot := 1; slot <= 2; slot++ {+ applyCounterClockHost(t, m, report(uint64(slot), slot, agent.libvirtDomain(slot, 0)))+ }++ omitted := applyCounterClockHost(t, m, report(3, 3))+ if omitted.Libvirt == nil || len(omitted.Libvirt.Domains) != 0 {+ t.Fatalf("readable inventory without the domain = %+v, want no domains", omitted.Libvirt)+ }++ returned := libvirtClockDomain(t, "returned", applyCounterClockHost(t, m, report(4, 4, agent.libvirtDomain(4, 0))))+ if returned.CPUUsageValid {+ t.Fatalf("domain absent from the previous report reported CPU usage %.1f", returned.CPUUsagePercent)+ }+ if !returned.IORatesValid {+ t.Fatal("returned domain lost its I/O rates")+ }+ wantTrueRate(t, "returned NetworkInRate", returned.NetworkInRate, libvirtClockNetInPerSecond)+ wantTrueRate(t, "returned DiskWriteRate", returned.DiskWriteRate, libvirtClockWritePerSecond)++ wantLibvirtTrueRates(t, "domain after its return", libvirtClockDomain(t, "slot 5", applyCounterClockHost(t, m, report(5, 5, agent.libvirtDomain(5, 0)))))++ unread := agent.hostReport("stream-a", 6, 6, agent.collectedAt(6))+ wantLibvirtTrueRates(t, "inventory preserved through a failed read", libvirtClockDomain(t, "slot 6", applyCounterClockHost(t, m, unread)))+ wantLibvirtTrueRates(t, "read after the failed one", libvirtClockDomain(t, "slot 7", applyCounterClockHost(t, m, report(7, 7, agent.libvirtDomain(7, 0)))))++ restarted := libvirtClockDomain(t, "restarted", applyCounterClockHost(t, m, report(8, 8, agent.libvirtDomain(8, 8))))+ if restarted.CPUUsageValid {+ t.Fatalf("restarted domain reported CPU usage %.1f across its counter reset", restarted.CPUUsagePercent)+ }+ if !restarted.IORatesValid || restarted.NetworkInRate != 0 || restarted.DiskReadRate != 0 {+ t.Fatalf("restarted domain I/O = %+v, want the counter-reset rebase of zero", restarted)+ }+ wantLibvirtTrueRates(t, "first interval after the restart", libvirtClockDomain(t, "slot 9", applyCounterClockHost(t, m, report(9, 9, agent.libvirtDomain(9, 8)))))+}++// Legacy reports have no ordered source clock, so their libvirt counters keep+// receipt time even when the inventory carries CollectedAt.+func TestApplyHostReportLegacyLibvirtRatesStayOnReceiptTime(t *testing.T) {+ m := newCounterClockHostMonitor(t)+ agent := newCounterClockAgent(time.Now().Add(-time.Hour))+ report := func(slot int) agentshost.Report {+ return agent.libvirtHostReport("", 0, slot, agent.collectedAt(slot), agent.libvirtCollectedAt(slot), agent.libvirtDomain(slot, 0))+ }+ applyCounterClockHost(t, m, report(1))+ sentAt := time.Now()+ domain := libvirtClockDomain(t, "legacy", applyCounterClockHost(t, m, report(2)))+ wall := time.Since(sentAt).Seconds()+ if !domain.CPUUsageValid || !domain.IORatesValid {+ t.Fatalf("legacy domain lost its receipt-time rates: %+v", domain)+ }+ delta := float64(agent.libvirtCounter(libvirtClockNetInPerSecond, 2, 0) - agent.libvirtCounter(libvirtClockNetInPerSecond, 1, 0))+ if domain.NetworkInRate < delta/wall {+ t.Fatalf("legacy NetworkInRate = %.1f, want the receipt-interval rate of at least %.1f (CollectedAt would give %.1f)",+ domain.NetworkInRate, delta/wall, delta/agent.interval.Seconds())+ }+}diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.goindex 31b0962645..ee784c4222 100644--- a/internal/monitoring/canonical_guardrails_test.go+++ b/internal/monitoring/canonical_guardrails_test.go@@ -2133,8 +2133,9 @@ func TestHostPhysicalDiskIOMetricsUseCanonicalDiskHistoryPath(t *testing.T) {}source := string(data)requiredSnippets := []string{- "m.writeHostPhysicalDiskIOMetrics(host, now)",- "func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time) {",+ "m.writeHostPhysicalDiskIOMetrics(host, now, counterSampled)",+ "func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time, sampled counterSampleTime) {",+ "readRate, writeRate, busyPct, _, _ := m.rateTracker.calculateSampledRates(trackerKey, current, sampled)",`resourceID := unifiedresources.HostSMARTDiskMetricID(host, disk)`,`m.metricsHistory.AddDiskMetric(resourceID, "diskread", readRate, now)`,`m.metricsStore.Write("disk", resourceID, "diskwrite", writeRate, now)`,diff --git a/internal/monitoring/issue1595_collection_trust_test.go b/internal/monitoring/issue1595_collection_trust_test.goindex 79f5366ca5..1ab373fc16 100644--- a/internal/monitoring/issue1595_collection_trust_test.go+++ b/internal/monitoring/issue1595_collection_trust_test.go@@ -434,7 +434,7 @@ func TestIdentitylessControllerMembersReadTheirWritersHistory(t *testing.T) {hosts := []models.Host{standalone, linked}for _, host := range hosts {- m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second))+ m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second), counterSampleTime{})}for _, host := range hosts {for i := range host.DiskIO {@@ -442,7 +442,7 @@ func TestIdentitylessControllerMembersReadTheirWritersHistory(t *testing.T) {host.DiskIO[i].WriteBytes += 1 << 20host.DiskIO[i].IOTime += 1000}- m.writeHostPhysicalDiskIOMetrics(host, now)+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampleTime{})m.writeHostSMARTMetrics(host, now)}m.writeSMARTMetrics(pveMember, now)@@ -638,13 +638,13 @@ func TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports(t *testing.T) {for _, device := range []string{"sdb", "sdc", "sdd", "sde", "sdf"} {host.DiskIO = append(host.DiskIO, models.DiskIO{Device: device, ReadBytes: 1 << 20, WriteBytes: 1 << 20, IOTime: 100})}- m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second))+ m.writeHostPhysicalDiskIOMetrics(host, now.Add(-30*time.Second), counterSampleTime{})for i := range host.DiskIO {host.DiskIO[i].ReadBytes += 1 << 20host.DiskIO[i].WriteBytes += 1 << 20host.DiskIO[i].IOTime += 1000}- m.writeHostPhysicalDiskIOMetrics(host, now)+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampleTime{})m.writeHostSMARTMetrics(host, now)store.Flush()diff --git a/internal/monitoring/monitor_agents.go b/internal/monitoring/monitor_agents.goindex ff42a8a809..ed04ec56f0 100644--- a/internal/monitoring/monitor_agents.go+++ b/internal/monitoring/monitor_agents.go@@ -226,6 +226,46 @@ func advanceHostReportOrder(return order, accepted}+// agentAuthoredCounterClock names the reporting machine's own clock, on which+// an agent authors each report's collection time.+const agentAuthoredCounterClock = "agent-authored"++// agentCounterSampleTime returns when an accepted agent report's cumulative+// counters were collected. The stream watermark admits sequenced reports in+// order, so the authored times of consecutive accepted reports measure the+// real collection interval, across an agent restart on the same machine clock+// too, even when reconnect delivery drains a buffered FIFO back to back and+// Pulse receipt times collapse toward zero. Legacy and untimestamped reports+// have no ordered source clock and keep receipt time; a switch between the two+// starts a new counter baseline.+func agentCounterSampleTime(sequenceID string, authoredAt time.Time) counterSampleTime {+ if _, _, sequenced := agentshost.ParseReportSequenceID(sequenceID); !sequenced || authoredAt.IsZero() {+ return counterSampleTime{}+ }+ return counterSampleTime{clock: agentAuthoredCounterClock, at: authoredAt.UTC()}+}++// agentIOCounterReadClock is the sampling mode of a host report whose agent+// stamped when it read the DiskIO and Network counters. It shares the machine+// clock with report Timestamp but not its offset from the read, so a switch+// between the two modes starts a new counter baseline.+const agentIOCounterReadClock = "agent-io-counter-read"++// agentIOCounterSampleTime returns when an accepted host report's DiskIO and+// Network counters were read. A current agent stamps the read itself, and the+// report Timestamp follows every optional collector, so the stamp is preferred.+// Older agents, and a missing or zero stamp, keep the report-Timestamp+// fallback, and legacy reports keep receipt time.+func agentIOCounterSampleTime(report agentshost.Report) counterSampleTime {+ if report.IOCountersCollectedAt != nil {+ if sampled := agentCounterSampleTime(report.SequenceID, *report.IOCountersCollectedAt); sampled.clock != "" {+ sampled.clock = agentIOCounterReadClock+ return sampled+ }+ }+ return agentCounterSampleTime(report.SequenceID, report.Timestamp)+}+// lockHostReportApplication serializes the complete state transition for one// host while allowing different hosts to ingest concurrently. Reserving a// sequence number without holding this lock through the subsequent state write@@ -2350,8 +2390,12 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *condockerVersion = runtimeVersion}+ counterSampled := agentCounterSampleTime(report.SequenceID, report.Timestamp)containers := make([]models.DockerContainer, 0, len(report.Containers))- for _, payload := range report.Containers {+ // Legacy container fields remain numeric. Carry the current calculation's+ // validity through to history so warmup and clock faults cannot become idle.+ containerNetworkValidity := make([]models.IORateValidity, len(report.Containers))+ for i, payload := range report.Containers {container := models.DockerContainer{ID: payload.ID,Name: payload.Name,@@ -2451,9 +2495,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *conmetrics.DiskWrite = clampToInt64(payload.BlockIO.WriteBytes)}- readRate, writeRate, netInRate, netOutRate := m.rateTracker.CalculateRates(+ readRate, writeRate, _, netInRate, netOutRate := m.rateTracker.calculateSampledRates(fmt.Sprintf("docker:%s:%s", identifier, containerIdentifier),metrics,+ counterSampled,)if container.BlockIO != nil && readRate >= 0 {@@ -2466,9 +2511,11 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con}if netInRate >= 0 {container.NetInRate = netInRate+ containerNetworkValidity[i].NetworkIn = true}if netOutRate >= 0 {container.NetOutRate = netOutRate+ containerNetworkValidity[i].NetworkOut = true}}@@ -2722,7 +2769,7 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *con}// Use a prefixed key (docker:containerID) to distinguish from Proxmox containers.- for _, container := range containers {+ for i, container := range containers {if container.ID == "" {continue}@@ -2743,10 +2790,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *conif m.metricsHistory != nil {m.metricsHistory.AddGuestMetric(metricKey, "cpu", models.DockerContainerCPUCapacityPercent(container, host.CPUs), now)m.metricsHistory.AddGuestMetric(metricKey, "memory", container.MemoryPercent, now)- if container.NetInRate >= 0 {+ if containerNetworkValidity[i].NetworkIn {m.metricsHistory.AddGuestMetric(metricKey, "netin", container.NetInRate, now)}- if container.NetOutRate >= 0 {+ if containerNetworkValidity[i].NetworkOut {m.metricsHistory.AddGuestMetric(metricKey, "netout", container.NetOutRate, now)}if diskReadRate >= 0 {@@ -2760,10 +2807,10 @@ func (m *Monitor) ApplyDockerReport(report agentsdocker.Report, tokenRecord *conif m.metricsStore != nil {m.metricsStore.Write("dockerContainer", container.ID, "cpu", models.DockerContainerCPUCapacityPercent(container, host.CPUs), now)m.metricsStore.Write("dockerContainer", container.ID, "memory", container.MemoryPercent, now)- if container.NetInRate >= 0 {+ if containerNetworkValidity[i].NetworkIn {m.metricsStore.Write("dockerContainer", container.ID, "netin", container.NetInRate, now)}- if container.NetOutRate >= 0 {+ if containerNetworkValidity[i].NetworkOut {m.metricsStore.Write("dockerContainer", container.ID, "netout", container.NetOutRate, now)}if diskReadRate >= 0 {@@ -3542,6 +3589,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.identifier,previousHostModel,observedAt,+ agentLibvirtSampleTime(report.SequenceID, report.Libvirt),)xcpngData := normalizeAgentXCPNGInventory(report.XCPNG, previousHostModel, observedAt)@@ -3744,6 +3792,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.// Network and disk bytes from the agent are cumulative totals since boot;// the RateTracker converts them to bytes/second, just like VMs and containers.now := receivedAt+ counterSampled := agentIOCounterSampleTime(report)var totalRXBytes, totalTXBytes uint64for _, nic := range host.NetworkInterfaces {@@ -3767,7 +3816,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.NetworkOut: int64(totalTXBytes),Timestamp: now,}- diskReadRate, diskWriteRate, netInRate, netOutRate := m.rateTracker.CalculateRates(hostRateKey, currentMetrics)+ diskReadRate, diskWriteRate, _, netInRate, netOutRate := m.rateTracker.calculateSampledRates(hostRateKey, currentMetrics, counterSampled)// Store computed rates on the host model so they flow through to unified resourcesif netInRate >= 0 {@@ -3878,7 +3927,7 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.}}- m.writeHostPhysicalDiskIOMetrics(host, now)+ m.writeHostPhysicalDiskIOMetrics(host, now, counterSampled)if m.metricsStore != nil {m.metricsStore.Write("agent", host.ID, "cpu", host.CPUUsage, now)@@ -4398,7 +4447,10 @@ func hostDiskIOMetricResourceID(host models.Host, io models.DiskIO, proxmoxDisksreturn fmt.Sprintf("%s:%s", strings.TrimSpace(host.ID), device)}-func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time) {+// writeHostPhysicalDiskIOMetrics records per-device disk rates. now is Pulse+// receipt time for history and rate-series freshness; sampled times the+// counter interval itself.+func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.Time, sampled counterSampleTime) {if shouldSkipNativeMockStateMetricWrites() || (m.metricsHistory == nil && m.metricsStore == nil) {return}@@ -4429,7 +4481,7 @@ func (m *Monitor) writeHostPhysicalDiskIOMetrics(host models.Host, now time.TimeDiskBusy: int64(io.IOTime),Timestamp: now,}- readRate, writeRate, busyPct, _, _ := m.rateTracker.CalculateRatesWithBusy(trackerKey, current)+ readRate, writeRate, busyPct, _, _ := m.rateTracker.calculateSampledRates(trackerKey, current, sampled)if readRate >= 0 {if m.metricsHistory != nil {diff --git a/internal/monitoring/monitor_libvirt.go b/internal/monitoring/monitor_libvirt.goindex 6dd306d60d..2ca0f30363 100644--- a/internal/monitoring/monitor_libvirt.go+++ b/internal/monitoring/monitor_libvirt.go@@ -18,11 +18,28 @@ const (maxAgentLibvirtDomainNameBytes = 256)+// agentLibvirtSampleTime returns when a libvirt inventory's domain counters+// were sampled. The agent stamps CollectedAt on its own clock just before it+// reads the domain stats, nearer that sample than the report's+// end-of-collection Timestamp. As with agentCounterSampleTime, only a+// sequenced report is ordered on that clock; legacy reports and inventories+// without CollectedAt keep receipt time.+func agentLibvirtSampleTime(sequenceID string, inventory *agentshost.LibvirtInventory) counterSampleTime {+ if inventory == nil {+ return counterSampleTime{}+ }+ return agentCounterSampleTime(sequenceID, inventory.CollectedAt)+}++// normalizeAgentLibvirtInventory admits a host report's libvirt inventory.+// observedAt is Pulse receipt time for freshness; sampled times the CPU and+// I/O counter intervals.func (m *Monitor) normalizeAgentLibvirtInventory(report *agentshost.LibvirtInventory,hostID string,previous *models.Host,observedAt time.Time,+ sampled counterSampleTime,) *models.HostLibvirtInventory {if report == nil {if previous != nil && previous.Libvirt != nil {@@ -34,13 +51,9 @@ func (m *Monitor) normalizeAgentLibvirtInventory(}previousByID := make(map[string]models.HostLibvirtDomain)- previousAt := time.Time{}- if previous != nil {- previousAt = previous.LastSeen- if previous.Libvirt != nil {- for _, domain := range previous.Libvirt.Domains {- previousByID[domain.ID] = domain- }+ if previous != nil && previous.Libvirt != nil {+ for _, domain := range previous.Libvirt.Domains {+ previousByID[domain.ID] = domain}}@@ -80,17 +93,24 @@ func (m *Monitor) normalizeAgentLibvirtInventory(domain.MemoryCurrentBytes = domain.MemoryMaximumBytes}- if prior, ok := previousByID[domain.ID]; ok {- domain.CPUUsagePercent, domain.CPUUsageValid = libvirtCPUUsagePercent(- prior,- domain,- observedAt.Sub(previousAt),+ if m != nil && m.rateTracker != nil {+ seriesID := "libvirt:" + strings.TrimSpace(hostID) + ":" + domain.ID+ // CPU time is sampled with the domain's I/O counters, so it is+ // measured over the same interval on the same clock. Every sample+ // moves its baseline; only a domain present in the previous report+ // whose CPU time did not reset yields a usage.+ cpuRate := m.rateTracker.calculateSampledCounterRate(+ seriesID+":cpu",+ domain.CPUTimeNanoseconds,+ observedAt,+ sampled,)- }+ if prior, ok := previousByID[domain.ID]; ok && domain.CPUTimeNanoseconds >= prior.CPUTimeNanoseconds {+ domain.CPUUsagePercent, domain.CPUUsageValid = libvirtCPUUsagePercent(cpuRate, domain.VCPUs)+ }- if m != nil && m.rateTracker != nil {- diskRead, diskWrite, netIn, netOut := m.rateTracker.CalculateRates(- "libvirt:"+strings.TrimSpace(hostID)+":"+domain.ID,+ diskRead, diskWrite, _, netIn, netOut := m.rateTracker.calculateSampledRates(+ seriesID,IOMetrics{DiskRead: uint64ToInt64(domain.DiskReadBytes),DiskWrite: uint64ToInt64(domain.DiskWriteBytes),@@ -98,6 +118,7 @@ func (m *Monitor) normalizeAgentLibvirtInventory(NetworkOut: uint64ToInt64(domain.NetworkTXBytes),Timestamp: observedAt,},+ sampled,)if diskRead >= 0 && diskWrite >= 0 && netIn >= 0 && netOut >= 0 {domain.DiskReadRate = diskRead@@ -142,16 +163,17 @@ func agentLibvirtDomainID(name string) string {return "domain-" + hex.EncodeToString(sum[:8])}-func libvirtCPUUsagePercent(previous, current models.HostLibvirtDomain, elapsed time.Duration) (float64, bool) {- if elapsed <= 0 || current.CPUTimeNanoseconds < previous.CPUTimeNanoseconds {+// libvirtCPUUsagePercent converts a domain's CPU-time rate, in nanoseconds per+// second summed across its vCPUs, to a percentage of its vCPU capacity. A+// negative rate means no comparable interval.+func libvirtCPUUsagePercent(cpuNanosecondsPerSecond float64, vcpus int) (float64, bool) {+ if cpuNanosecondsPerSecond < 0 {return 0, false}- vcpus := current.VCPUsif vcpus <= 0 {vcpus = 1}- delta := current.CPUTimeNanoseconds - previous.CPUTimeNanoseconds- percent := (float64(delta) / float64(elapsed.Nanoseconds()) / float64(vcpus)) * 100+ percent := (cpuNanosecondsPerSecond / float64(time.Second) / float64(vcpus)) * 100if math.IsNaN(percent) || math.IsInf(percent, 0) {return 0, false}diff --git a/internal/monitoring/monitor_libvirt_test.go b/internal/monitoring/monitor_libvirt_test.goindex cbf79becc8..ebc5c072b1 100644--- a/internal/monitoring/monitor_libvirt_test.go+++ b/internal/monitoring/monitor_libvirt_test.go@@ -24,7 +24,7 @@ func TestNormalizeAgentLibvirtInventoryDerivesCPUAndIORates(t *testing.T) {DiskReadBytes: 3000,DiskWriteBytes: 4000,}}}- first := monitor.normalizeAgentLibvirtInventory(firstReport, "host-1", nil, t0)+ first := monitor.normalizeAgentLibvirtInventory(firstReport, "host-1", nil, t0, counterSampleTime{})if first == nil || len(first.Domains) != 1 {t.Fatalf("first inventory = %+v", first)}@@ -49,7 +49,7 @@ func TestNormalizeAgentLibvirtInventoryDerivesCPUAndIORates(t *testing.T) {DiskReadBytes: 6000,DiskWriteBytes: 8000,}}}- second := monitor.normalizeAgentLibvirtInventory(secondReport, "host-1", previous, t0.Add(10*time.Second))+ second := monitor.normalizeAgentLibvirtInventory(secondReport, "host-1", previous, t0.Add(10*time.Second), counterSampleTime{})if second == nil || len(second.Domains) != 1 {t.Fatalf("second inventory = %+v", second)}@@ -76,7 +76,7 @@ func TestNormalizeAgentLibvirtInventoryBoundsAndDeduplicates(t *testing.T) {{Name: "vm", State: "running"},{Name: "bad\x00name", State: "running"},}}- got := monitor.normalizeAgentLibvirtInventory(report, "host-1", nil, time.Now())+ got := monitor.normalizeAgentLibvirtInventory(report, "host-1", nil, time.Now(), counterSampleTime{})if got == nil || len(got.Domains) != 1 {t.Fatalf("inventory = %+v, want one domain", got)}@@ -99,6 +99,7 @@ func TestNormalizeAgentLibvirtInventoryPreservesLastSuccessfulSampleOnFailure(t"host-1",previous,time.Now(),+ counterSampleTime{},)if got == nil || len(got.Domains) != 1 || !got.CollectedAt.Equal(collectedAt) {t.Fatalf("preserved inventory = %+v", got)diff --git a/internal/monitoring/ratetracker.go b/internal/monitoring/ratetracker.goindex 13a59cd8c2..3c5ecbfe94 100644--- a/internal/monitoring/ratetracker.go+++ b/internal/monitoring/ratetracker.go@@ -10,20 +10,41 @@ import (// IOMetrics is an alias for models.IOMetrics.type IOMetrics = models.IOMetrics-type counterBaseline struct {- value int64+type counterBaseline[T int64 | uint64] struct {+ value TobservedAt time.Timeinitialized bool}type counterHistory struct {- diskRead counterBaseline- diskWrite counterBaseline- diskBusy counterBaseline- networkIn counterBaseline- networkOut counterBaseline+ diskRead counterBaseline[int64]+ diskWrite counterBaseline[int64]+ diskBusy counterBaseline[int64]+ networkIn counterBaseline[int64]+ networkOut counterBaseline[int64]+ cpuTime counterBaseline[uint64]lastObservedAt time.TimesourceUptime uint64+ sampleClock string+}++// counterSampleTime is when a producer sampled a set of cumulative counters,+// on a named clock whose samples the producer has already ordered. Rates+// measure intervals on that clock, while IOMetrics.Timestamp stays Pulse+// receipt time for freshness and cleanup. The zero value keeps receipt time+// for both, which pull-based collectors observe directly.+type counterSampleTime struct {+ clock string+ at time.Time+}++// counterTime returns when a counter was sampled: the named clock's time, or+// else the counter's own receipt time with the payload receipt as fallback.+func (sampled counterSampleTime) counterTime(counterReceivedAt, receivedAt time.Time) time.Time {+ if sampled.clock != "" {+ return sampled.at+ }+ return observationTime(counterReceivedAt, receivedAt)}// RateTracker converts cumulative byte counters into adjacent-sample rates.@@ -41,8 +62,9 @@ func NewRateTracker() *RateTracker {// CalculateRates calculates disk and network rates in bytes per second.// A negative result means the upstream counter was absent, the sample was-// out-of-order, or no earlier observation exists. A returned zero is a valid-// observed idle/reset interval.+// out-of-order, no earlier observation exists, or no comparable interval+// exists on the sample's clock. A returned zero is a valid observed+// idle/reset interval.func (rt *RateTracker) CalculateRates(guestID string, current IOMetrics) (diskReadRate, diskWriteRate, netInRate, netOutRate float64) {diskReadRate, diskWriteRate, _, netInRate, netOutRate = rt.calculateRates(guestID, current)return@@ -55,18 +77,20 @@ func (rt *RateTracker) CalculateRatesWithBusy(guestID string, current IOMetrics)}func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskReadRate, diskWriteRate, diskBusyPct, netInRate, netOutRate float64) {+ return rt.calculateSampledRates(guestID, current, counterSampleTime{})+}++// calculateSampledRates measures each counter interval on sampled's clock.+// Samples on a different clock than the baseline start a new counter epoch,+// because their times cannot be subtracted from each other.+func (rt *RateTracker) calculateSampledRates(guestID string, current IOMetrics, sampled counterSampleTime) (diskReadRate, diskWriteRate, diskBusyPct, netInRate, netOutRate float64) {rt.mu.Lock()defer rt.mu.Unlock()if current.Timestamp.IsZero() {return -1, -1, -1, -1, -1}-- history := rt.history[guestID]- if history == nil {- history = &counterHistory{}- rt.history[guestID] = history- }+ history, sampled := rt.sampledHistory(guestID, current.Timestamp, sampled)if current.SourceUptime > 0 {if history.sourceUptime > 0 && current.SourceUptime < history.sourceUptime {history.resetCounterEpoch()@@ -75,11 +99,12 @@ func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskRe}presence := current.Presence.Effective()- diskReadRate = calculateCounterRate(&history.diskRead, current.DiskRead, observationTime(current.ObservedAt.DiskRead, current.Timestamp), presence.DiskRead)- diskWriteRate = calculateCounterRate(&history.diskWrite, current.DiskWrite, observationTime(current.ObservedAt.DiskWrite, current.Timestamp), presence.DiskWrite)- diskBusyRate := calculateCounterRate(&history.diskBusy, current.DiskBusy, observationTime(current.ObservedAt.DiskBusy, current.Timestamp), presence.DiskBusy)- netInRate = calculateCounterRate(&history.networkIn, current.NetworkIn, observationTime(current.ObservedAt.NetworkIn, current.Timestamp), presence.NetworkIn)- netOutRate = calculateCounterRate(&history.networkOut, current.NetworkOut, observationTime(current.ObservedAt.NetworkOut, current.Timestamp), presence.NetworkOut)+ ordered := sampled.clock != ""+ diskReadRate = sampledCounterRate(&history.diskRead, current.DiskRead, sampled.counterTime(current.ObservedAt.DiskRead, current.Timestamp), presence.DiskRead, ordered)+ diskWriteRate = sampledCounterRate(&history.diskWrite, current.DiskWrite, sampled.counterTime(current.ObservedAt.DiskWrite, current.Timestamp), presence.DiskWrite, ordered)+ diskBusyRate := sampledCounterRate(&history.diskBusy, current.DiskBusy, sampled.counterTime(current.ObservedAt.DiskBusy, current.Timestamp), presence.DiskBusy, ordered)+ netInRate = sampledCounterRate(&history.networkIn, current.NetworkIn, sampled.counterTime(current.ObservedAt.NetworkIn, current.Timestamp), presence.NetworkIn, ordered)+ netOutRate = sampledCounterRate(&history.networkOut, current.NetworkOut, sampled.counterTime(current.ObservedAt.NetworkOut, current.Timestamp), presence.NetworkOut, ordered)if diskBusyRate < 0 {diskBusyPct = -1@@ -92,18 +117,51 @@ func (rt *RateTracker) calculateRates(guestID string, current IOMetrics) (diskRe}}- if current.Timestamp.After(history.lastObservedAt) {- history.lastObservedAt = current.Timestamp- }return}+// sampledHistory applies clock epochs and receipt-time cleanup to every+// counter series. The caller holds rt.mu and supplies a nonzero receipt time.+func (rt *RateTracker) sampledHistory(seriesID string, receivedAt time.Time, sampled counterSampleTime) (*counterHistory, counterSampleTime) {+ if sampled.at.IsZero() {+ sampled = counterSampleTime{}+ }+ history := rt.history[seriesID]+ if history == nil {+ history = &counterHistory{}+ rt.history[seriesID] = history+ }+ if sampled.clock != history.sampleClock {+ history.resetCounterEpoch()+ history.sampleClock = sampled.clock+ }+ if receivedAt.After(history.lastObservedAt) {+ history.lastObservedAt = receivedAt+ }+ return history, sampled+}++// calculateSampledCounterRate measures one cumulative counter outside the disk+// and network set, such as a libvirt domain's CPU time, under the same clock,+// baseline and cleanup rules as calculateSampledRates. Preserve its unsigned+// range: cumulative CPU nanoseconds can exceed the signed byte-counter range.+func (rt *RateTracker) calculateSampledCounterRate(seriesID string, value uint64, receivedAt time.Time, sampled counterSampleTime) float64 {+ rt.mu.Lock()+ defer rt.mu.Unlock()+ if receivedAt.IsZero() {+ return -1+ }+ history, sampled := rt.sampledHistory(seriesID, receivedAt, sampled)+ return sampledCounterRate(&history.cpuTime, value, sampled.counterTime(time.Time{}, receivedAt), true, sampled.clock != "")+}+func (history *counterHistory) resetCounterEpoch() {- history.diskRead = counterBaseline{}- history.diskWrite = counterBaseline{}- history.diskBusy = counterBaseline{}- history.networkIn = counterBaseline{}- history.networkOut = counterBaseline{}+ history.diskRead = counterBaseline[int64]{}+ history.diskWrite = counterBaseline[int64]{}+ history.diskBusy = counterBaseline[int64]{}+ history.networkIn = counterBaseline[int64]{}+ history.networkOut = counterBaseline[int64]{}+ history.cpuTime = counterBaseline[uint64]{}}func observationTime(counterTime, fallback time.Time) time.Time {@@ -113,7 +171,7 @@ func observationTime(counterTime, fallback time.Time) time.Time {return counterTime}-func calculateCounterRate(baseline *counterBaseline, value int64, observedAt time.Time, present bool) float64 {+func calculateCounterRate[T int64 | uint64](baseline *counterBaseline[T], value T, observedAt time.Time, present bool) float64 {if !present {return -1}@@ -140,6 +198,19 @@ func calculateCounterRate(baseline *counterBaseline, value int64, observedAt timreturn float64(value-previous) / elapsed}+// sampledCounterRate is calculateCounterRate for a sample that may sit on an+// ordered clock. There a time that does not advance past the baseline is a+// clock step or a malformed time, not a late sample, so neither sample bounds+// a known interval: the baseline is dropped, this sample is unknown, and the+// next sample starts a new baseline.+func sampledCounterRate[T int64 | uint64](baseline *counterBaseline[T], value T, observedAt time.Time, present, ordered bool) float64 {+ if ordered && present && baseline.initialized && !observedAt.After(baseline.observedAt) {+ *baseline = counterBaseline[T]{}+ return -1+ }+ return calculateCounterRate(baseline, value, observedAt, present)+}+// Clear removes all tracked data.func (rt *RateTracker) Clear() {rt.mu.Lock()diff --git a/pkg/agents/host/report.go b/pkg/agents/host/report.goindex e9bab8028d..64b7486268 100644--- a/pkg/agents/host/report.go+++ b/pkg/agents/host/report.go@@ -65,6 +65,11 @@ type Report struct {Tags []string `json:"tags,omitempty"`Timestamp time.Time `json:"timestamp"`SequenceID string `json:"sequenceId,omitempty"`+ // IOCountersCollectedAt is when the agent read the cumulative DiskIO and+ // Network counters, on the same clock as Timestamp. Timestamp is stamped+ // after every optional collector, so only this bounds the counter read.+ // Optional: older agents omit it.+ IOCountersCollectedAt *time.Time `json:"ioCountersCollectedAt,omitempty"`}// LibvirtInventory is the bounded, read-only domain inventory collected fromdiff --git a/pkg/agents/host/report_io_counters_collected_at_test.go b/pkg/agents/host/report_io_counters_collected_at_test.gonew file mode 100644index 0000000000..9faa966bf4--- /dev/null+++ b/pkg/agents/host/report_io_counters_collected_at_test.go@@ -0,0 +1,47 @@+package host++import (+ "bytes"+ "encoding/json"+ "testing"+ "time"+)++// IOCountersCollectedAt is optional on the wire: an older agent's report+// decodes without it, a report without it emits nothing new, and a stamped+// report keeps it apart from Timestamp through the JSON that both delivery+// and the agent's persisted report buffer use.+func TestReportIOCountersCollectedAtIsOptionalOnTheWire(t *testing.T) {+ var older Report+ if err := json.Unmarshal([]byte(`{"agent":{"id":"agent-a"},"host":{"hostname":"host-a"},"timestamp":"2026-10-30T12:00:50Z","sequenceId":"v1:stream-a:2"}`), &older); err != nil {+ t.Fatalf("decode older report: %v", err)+ }+ if older.IOCountersCollectedAt != nil {+ t.Fatalf("older report decoded a counter read stamp: %v", older.IOCountersCollectedAt)+ }++ unstamped, err := json.Marshal(Report{Timestamp: time.Date(2026, 10, 30, 12, 0, 50, 0, time.UTC)})+ if err != nil {+ t.Fatalf("encode unstamped report: %v", err)+ }+ if bytes.Contains(unstamped, []byte("ioCountersCollectedAt")) {+ t.Fatalf("unstamped report emitted the optional field: %s", unstamped)+ }++ readAt := time.Date(2026, 10, 30, 12, 0, 30, 123456789, time.UTC)+ stamped := Report{Timestamp: readAt.Add(20 * time.Second), IOCountersCollectedAt: &readAt}+ raw, err := json.Marshal(stamped)+ if err != nil {+ t.Fatalf("encode stamped report: %v", err)+ }+ var decoded Report+ if err := json.Unmarshal(raw, &decoded); err != nil {+ t.Fatalf("decode stamped report: %v", err)+ }+ if decoded.IOCountersCollectedAt == nil || !decoded.IOCountersCollectedAt.Equal(readAt) {+ t.Fatalf("counter read stamp = %v, want %v", decoded.IOCountersCollectedAt, readAt)+ }+ if !decoded.Timestamp.Equal(stamped.Timestamp) {+ t.Fatalf("Timestamp = %v, want the report's own %v", decoded.Timestamp, stamped.Timestamp)+ }+}diff --git a/scripts/release_control/canonical_completion_guard_test.py b/scripts/release_control/canonical_completion_guard_test.pyindex 040d162608..632e3a7360 100644--- a/scripts/release_control/canonical_completion_guard_test.py+++ b/scripts/release_control/canonical_completion_guard_test.py@@ -256,9 +256,13 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"allow_same_subsystem_tests": False,"test_prefixes": [],"exact_files": [+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/api/host_agent_removal_lifecycle_integration_test.go","internal/config/host_continuity_test.go","internal/models/metrics_types_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/availability_probe_agent_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go",@@ -447,9 +451,13 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"allow_same_subsystem_tests": False,"test_prefixes": [],"exact_files": [+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/api/host_agent_removal_lifecycle_integration_test.go","internal/config/host_continuity_test.go","internal/models/metrics_types_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/availability_probe_agent_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go",@@ -486,6 +494,9 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"test_prefixes": [],"exact_files": ["internal/config/host_continuity_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go","internal/monitoring/issue1485_unraid_lifecycle_test.go",@@ -495,6 +506,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"internal/monitoring/monitor_host_agents_test.go","internal/monitoring/monitor_package_updates_test.go","internal/unifiedresources/code_standards_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go",],}@@ -776,6 +788,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"test_prefixes": [],"exact_files": ["internal/agentupdate/coverage_test.go",+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/hostagent/action_runner_client_test.go","internal/hostagent/agent_flushbuffer_test.go","internal/hostagent/agent_metrics_test.go",@@ -795,6 +808,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):"internal/hostagent/smartctl_standby_guard_test.go","internal/hostagent/storage_cleanup_test.go","internal/hostagent/unraid_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go",],}diff --git a/scripts/release_control/subsystem_lookup_test.py b/scripts/release_control/subsystem_lookup_test.pyindex 61b94eb0f1..847cab70da 100644--- a/scripts/release_control/subsystem_lookup_test.py+++ b/scripts/release_control/subsystem_lookup_test.py@@ -4644,9 +4644,13 @@ class SubsystemLookupTest(unittest.TestCase):self.assertEqual(lifecycle_match["verification_requirement"]["exact_files"],[+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/api/host_agent_removal_lifecycle_integration_test.go","internal/config/host_continuity_test.go","internal/models/metrics_types_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/availability_probe_agent_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go",@@ -4675,6 +4679,9 @@ class SubsystemLookupTest(unittest.TestCase):monitoring_match["verification_requirement"]["exact_files"],["internal/config/host_continuity_test.go",+ "internal/monitoring/agent_report_counter_clock_test.go",+ "internal/monitoring/agent_report_io_counter_clock_test.go",+ "internal/monitoring/agent_report_libvirt_clock_test.go","internal/monitoring/docker_filesystem_evidence_test.go","internal/monitoring/docker_metric_presence_test.go","internal/monitoring/issue1485_unraid_lifecycle_test.go",@@ -4684,6 +4691,7 @@ class SubsystemLookupTest(unittest.TestCase):"internal/monitoring/monitor_host_agents_test.go","internal/monitoring/monitor_package_updates_test.go","internal/unifiedresources/code_standards_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go",],)@@ -4989,6 +4997,7 @@ class SubsystemLookupTest(unittest.TestCase):match["verification_requirement"]["exact_files"],["internal/agentupdate/coverage_test.go",+ "internal/api/agent_report_decode_io_stamp_compat_test.go","internal/hostagent/action_runner_client_test.go","internal/hostagent/agent_flushbuffer_test.go","internal/hostagent/agent_metrics_test.go",@@ -5008,6 +5017,7 @@ class SubsystemLookupTest(unittest.TestCase):"internal/hostagent/smartctl_standby_guard_test.go","internal/hostagent/storage_cleanup_test.go","internal/hostagent/unraid_test.go",+ "pkg/agents/host/report_io_counters_collected_at_test.go","pkg/agents/host/report_test.go",],)