Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Judge connection freshness by the monitor's live poll cadence

rcourtman912d7ceb51fb
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 6a8d9af826..ae067a9cff 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -9237,6 +9237,15 @@ registration, enrolment, install, update, removal and report identity are
unchanged; the per-image memo lives in the registry checker and is pruned
each collection cycle to the images in use.
+### Connection freshness cadence only
+
+`internal/api/connections_alerts.go` changed only so PVE, PBS and PMG
+connection rows scale their stale cutoff by the monitor's base poll cadence
+(`Monitor.BasePollInterval`, which honours runtime polling overrides) rather
+than by a non-default org's detached config copy. Agent connection rows keep
+their own heartbeat cutoff. Agent registration, enrolment, install, update,
+removal and report identity are unchanged.
+
### Host snapshots report a linked agent's own heartbeat
`internal/monitoring/monitor.go` changed only so a host produced from the read
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 0ec4e56316..53b2a60b58 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -9997,6 +9997,25 @@ metadata, but API freshness and `lastSeenAt` projections use the server-authored
receipt time supplied by monitoring. API consumers must not substitute the
agent clock for disconnect detection or apply a second, shorter generic
Connections staleness window.
+PVE, PBS and PMG connection rows scale their active-to-stale cutoff by the
+cadence the org's monitor polls at.
+`buildAggregatorInputsWithRuntimeSources` reads `Monitor.BasePollInterval`
+(clamped like the scheduler, so an out-of-range configured interval is judged
+by the cadence actually polled) whenever a monitor is present, and falls back
+to the request config only when there is no monitor. A non-default org's request config is its monitor's
+detached copy (#1619), which a system-settings save does not update; saved PBS
+and PMG intervals reach that monitor only as runtime overrides. Reading the
+copy left Settings > Infrastructure rows stale for part of every cycle after
+an interval was raised, and slow to go stale after one was lowered.
+Adaptive polling still scales by the larger of that base and
+`PlannedPollInterval`, a conservative floor that can exceed the adaptive
+cadence. The connections list, runtime inventory sources and
+diagnostics all share this input builder. The system settings GET is not
+affected: the router serves `/api/config/system` from `SystemSettingsHandler`,
+which reads the base config and saved settings, not a tenant copy.
+`TestContract_ConnectionFreshnessFollowsSavedPollingIntervalsInTenantMonitors`
+in `internal/api/contract_test.go` drives a real settings save into a tenant
+monitor built from a detached copy and checks both directions.
Availability target writes now use `observationLocationIds` as the canonical
bounded set. Values are source-owned IDs (`pulse:local` or
`agent:<agent-id>`), are normalized and deduplicated by the server, and every
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 5bb69533b2..58f47f21be 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -3610,6 +3610,30 @@ tighten it only when a freshly shortened interval justifies an earlier run.
Without this, a sixty-second availability target polls at the ten-second
tick cadence whenever adaptive polling is off, which is the default.
+### Base poll cadence is read through the monitor, not its config copy
+
+`Monitor.BasePollInterval` (`internal/monitoring/monitor_runtime_settings.go`)
+is the exported reader for the clamped base cadence the monitor polls PVE,
+PBS and PMG at. For the built-in poll providers it returns the same values as
+the scheduler's `baseIntervalForInstanceType`, including PBS and PMG runtime
+polling overrides, but skips the poll-provider lookup, so it never takes
+`m.mu`. A replacement provider's own `BaseInterval`, a provider's fixed
+instance interval, and a task already queued at its previous interval can
+differ from it until the next planning pass. A monitor without config reports
+zero where the scheduler falls back to ten seconds.
+Non-default tenant monitors poll against a detached config copy (#1619), so a
+saved PBS or PMG interval reaches them only as one of those overrides, and
+`GetConfig` keeps the old value until that monitor is rebuilt (a restart, or a
+full reload such as a PVE interval save). Code outside the monitor that
+judges poll freshness, starting with the connections list, must read the cadence
+through `BasePollInterval`, not from the config. It reports the same base under
+adaptive scheduling, where it is only a floor: the adaptive scheduler plans
+each instance's interval independently of the per-platform intervals, and
+`PlannedPollInterval` reports that plan. The fixed-cadence branch of
+`Monitor.resourcePollIntervals` reads the same accessor. Regression coverage:
+`TestMonitorBasePollIntervalReportsSchedulerBaseCadence` in
+`internal/monitoring/canonical_guardrails_test.go`.
+
### Host report admission does not wedge on stale removal blocks
Host-agent report admission consults removal blocks across the durable
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 34d4e0d1c7..f1bbc4f68a 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -6629,6 +6629,16 @@ to it.
Both are read-path changes to source freshness and presentation; no storage,
retention, backup or recovery path is added or moved.
+### Connection freshness cadence opens no storage or recovery path
+
+`internal/api/connections_alerts.go` now scales the connections list's
+active-to-stale cutoff by the monitor's base poll cadence
+(`Monitor.BasePollInterval`), which honours saved PBS and PMG intervals that
+reach a non-default org's monitor only as runtime overrides. This is a
+read-path change to connection freshness. Backup polling, PBS backup and
+datastore collection, retention and recovery are unchanged, and nothing new
+is persisted.
+
### Webhook configuration never persists the API mask
`internal/api/alerting/notifications.go` resolves each `***REDACTED***`
diff --git a/internal/api/connections_aggregator.go b/internal/api/connections_aggregator.go
index dd07879e2c..eb32d90d67 100644
--- a/internal/api/connections_aggregator.go
+++ b/internal/api/connections_aggregator.go
@@ -133,7 +133,9 @@ type aggregatorInputs struct {
pbsReportedNodeNames map[string]string
// Configured poll cadences, used to scale the active→stale cutoff so slow
- // cadences (e.g. 5 minutes) don't read as permanently stale. Zero means
+ // cadences (e.g. 5 minutes) don't read as permanently stale. With a
+ // monitor present they are its base cadences (Monitor.BasePollInterval,
+ // which honours runtime polling overrides), not the config copy. Zero means
// unknown and falls back to the connectionStaleThreshold floor. VMware and
// TrueNAS cadences come from their summaries/instances instead.
pvePollingInterval time.Duration
diff --git a/internal/api/connections_alerts.go b/internal/api/connections_alerts.go
index 7c2229a48e..0a8dbdbf41 100644
--- a/internal/api/connections_alerts.go
+++ b/internal/api/connections_alerts.go
@@ -66,6 +66,19 @@ func buildAggregatorInputsWithRuntimeSources(
inputs.instanceHealth = instanceHealthByKey(monitor.SchedulerHealth())
inputs.availabilityStatuses = monitor.AvailabilityStatusSnapshot()
inputs.pbsReportedNodeNames = pbsReportedNodeNamesByInstance(monitor.PBSInstancesSnapshot())
+ // A non-default org's cfg is its monitor's detached config copy
+ // (#1619), which a settings save does not update: saved PBS and PMG
+ // intervals reach the monitor only as runtime overrides. Scale the
+ // stale cutoff by the cadence the monitor actually polls at.
+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePVE); interval > 0 {
+ inputs.pvePollingInterval = interval
+ }
+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePBS); interval > 0 {
+ inputs.pbsPollingInterval = interval
+ }
+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePMG); interval > 0 {
+ inputs.pmgPollingInterval = interval
+ }
inputs.plannedPollIntervals = plannedPollIntervalsForConfig(monitor, cfg)
} else {
inputs.hosts = []models.Host{}
diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go
index 5f9d67b9f6..3f6c1b18ab 100644
--- a/internal/api/contract_test.go
+++ b/internal/api/contract_test.go
@@ -3581,6 +3581,97 @@ func TestContract_GenerateStyledMockSeries_UsesTimestampBasedCurve(t *testing.T)
}
}
+// Non-default organisations poll against a detached config copy (#1619). A
+// system-settings save updates the server's base config and pushes saved PBS
+// and PMG intervals into each live tenant monitor as runtime overrides, but
+// the copy that configapi hands that org's connections list (m.GetConfig())
+// keeps the old values. Connection freshness must scale by the cadence the
+// monitor polls at, or Settings > Infrastructure rows read stale for part of
+// every slower cycle.
+func TestContract_ConnectionFreshnessFollowsSavedPollingIntervalsInTenantMonitors(t *testing.T) {
+ setMockModeForTest(t, false)
+
+ cases := []struct {
+ name string
+ configured time.Duration
+ saved time.Duration
+ wantState ConnectionState
+ }{
+ {
+ name: "raised interval keeps an on-schedule poll active",
+ configured: time.Minute,
+ saved: 5 * time.Minute,
+ wantState: ConnectionStateActive,
+ },
+ {
+ name: "lowered interval judges the same poll stale",
+ configured: 5 * time.Minute,
+ saved: time.Minute,
+ wantState: ConnectionStateStale,
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ tempDir := t.TempDir()
+ base := &config.Config{
+ DataPath: tempDir,
+ ConfigPath: tempDir,
+ PVEPollingInterval: 10 * time.Second,
+ PBSPollingInterval: tc.configured,
+ PMGPollingInterval: tc.configured,
+ }
+ tenant, err := monitoring.New(base.DeepCopy())
+ if err != nil {
+ t.Fatalf("new tenant monitor: %v", err)
+ }
+ t.Cleanup(tenant.Stop)
+
+ tokenVal := "tenant-polling-token"
+ base.APITokens = []config.APITokenRecord{
+ {ID: "token1", Hash: authpkg.HashAPIToken(tokenVal), Name: "Test Token"},
+ }
+ handler := newTestSystemSettingsHandler(base, config.NewConfigPersistence(tempDir), tenant, func() {}, func() error { return nil })
+ seconds := int(tc.saved / time.Second)
+ body, _ := json.Marshal(map[string]any{"pbsPollingInterval": seconds, "pmgPollingInterval": seconds})
+ req := httptest.NewRequest(http.MethodPost, "/api/system-settings", bytes.NewReader(body))
+ req.Header.Set("X-API-Token", tokenVal)
+ rec := httptest.NewRecorder()
+ handler.HandleUpdateSystemSettings(rec, req)
+ if rec.Code != http.StatusOK {
+ t.Fatalf("save status = %d, body: %s", rec.Code, rec.Body.String())
+ }
+
+ cfg := tenant.GetConfig()
+ if base.PBSPollingInterval != tc.saved || cfg.PBSPollingInterval != tc.configured {
+ t.Fatalf("PBS interval base = %v, tenant copy = %v; want the save to reach only the base (%v) and the copy to keep %v",
+ base.PBSPollingInterval, cfg.PBSPollingInterval, tc.saved, tc.configured)
+ }
+
+ now := time.Now()
+ lastSuccess := now.Add(-4 * time.Minute)
+ inputs := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, tenant, aggregatorRuntimeSources{orgID: "tenant-a"})
+ inputs.now = now
+ inputs.pbsInstances = []config.PBSInstance{{Name: "backup", Host: "https://pbs.lab.local:8007"}}
+ inputs.pmgInstances = []config.PMGInstance{{Name: "mail", Host: "https://pmg.lab.local:8006"}}
+ inputs.instanceHealth = map[string]monitoring.InstanceHealth{
+ "pbs::backup": {PollStatus: monitoring.InstancePollStatus{LastSuccess: &lastSuccess}},
+ "pmg::mail": {PollStatus: monitoring.InstancePollStatus{LastSuccess: &lastSuccess}},
+ }
+
+ states := map[ConnectionType]ConnectionState{}
+ for _, conn := range buildConnections(inputs) {
+ states[conn.Type] = conn.State
+ }
+ for _, connType := range []ConnectionType{ConnectionTypePBS, ConnectionTypePMG} {
+ if got := states[connType]; got != tc.wantState {
+ t.Errorf("%s state = %q, want %q for a 4m-old poll after saving %v (tenant copy still %v)",
+ connType, got, tc.wantState, tc.saved, tc.configured)
+ }
+ }
+ })
+ }
+}
+
func TestContract_PlatformMockToggleRebindsRuntimeConnectionsAndResources(t *testing.T) {
setMockModeForTest(t, false)
diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go
index 4472c7f966..7d409f0fc0 100644
--- a/internal/monitoring/canonical_guardrails_test.go
+++ b/internal/monitoring/canonical_guardrails_test.go
@@ -2888,6 +2888,83 @@ func TestResourceStaleThresholdsPreserveDefaultFloors(t *testing.T) {
}
}
+// Consumers outside the monitor, such as the connections list, judge poll
+// freshness by Monitor.BasePollInterval. It must report the base cadence the
+// scheduler polls each platform at, including runtime polling overrides on a
+// detached tenant config copy (#1619), and must not take m.mu, which the
+// scheduler's poll-provider lookup takes.
+func TestMonitorBasePollIntervalReportsSchedulerBaseCadence(t *testing.T) {
+ newTenantMonitor := func(override time.Duration) *Monitor {
+ base := &config.Config{
+ PVEPollingInterval: 30 * time.Second,
+ PBSPollingInterval: time.Minute,
+ PMGPollingInterval: time.Minute,
+ }
+ monitor := &Monitor{config: base.DeepCopy()}
+ monitor.SetPBSPollingInterval(override)
+ monitor.SetPMGPollingInterval(override)
+ return monitor
+ }
+
+ t.Run("runtime overrides on a detached copy, clamped like the scheduler", func(t *testing.T) {
+ for override, want := range map[time.Duration]time.Duration{
+ 5 * time.Second: 10 * time.Second,
+ 90 * time.Second: 90 * time.Second,
+ 5 * time.Minute: 5 * time.Minute,
+ 2 * time.Hour: time.Hour,
+ } {
+ monitor := newTenantMonitor(override)
+ for _, instanceType := range []InstanceType{InstanceTypePBS, InstanceTypePMG} {
+ if got := monitor.BasePollInterval(instanceType); got != want {
+ t.Errorf("%s base interval = %v for override %v, want %v", instanceType, got, override, want)
+ }
+ }
+ for _, instanceType := range []InstanceType{InstanceTypePVE, InstanceTypePBS, InstanceTypePMG} {
+ if got, scheduled := monitor.BasePollInterval(instanceType), monitor.baseIntervalForInstanceType(instanceType); got != scheduled {
+ t.Errorf("%s base interval = %v for override %v, want the scheduler's %v", instanceType, got, override, scheduled)
+ }
+ }
+ }
+ })
+
+ t.Run("adaptive scheduling keeps the same base", func(t *testing.T) {
+ monitor := newTenantMonitor(5 * time.Minute)
+ monitor.scheduler = NewAdaptiveScheduler(SchedulerConfig{}, nil, nil, nil)
+ if got := monitor.BasePollInterval(InstanceTypePBS); got != 5*time.Minute {
+ t.Fatalf("PBS base interval = %v under adaptive scheduling, want the saved %v", got, 5*time.Minute)
+ }
+ })
+
+ t.Run("does not take the monitor lock", func(t *testing.T) {
+ monitor := newTenantMonitor(5 * time.Minute)
+ monitor.mu.Lock()
+ defer monitor.mu.Unlock()
+ done := make(chan time.Duration, 1)
+ go func() { done <- monitor.BasePollInterval(InstanceTypePBS) }()
+ select {
+ case got := <-done:
+ if got != 5*time.Minute {
+ t.Fatalf("PBS base interval = %v, want %v", got, 5*time.Minute)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("BasePollInterval blocked while m.mu was held")
+ }
+ })
+
+ t.Run("unknown cadence reads zero", func(t *testing.T) {
+ var nilMonitor *Monitor
+ for name, got := range map[string]time.Duration{
+ "nil monitor": nilMonitor.BasePollInterval(InstanceTypePBS),
+ "monitor no config": (&Monitor{}).BasePollInterval(InstanceTypePVE),
+ "availability": newTenantMonitor(time.Minute).BasePollInterval(InstanceTypeAvailability),
+ } {
+ if got != 0 {
+ t.Errorf("%s base interval = %v, want 0", name, got)
+ }
+ }
+ })
+}
+
// Non-default tenant monitors poll against a detached config copy (#1619), so
// a saved PBS or PMG interval reaches them only as a runtime override. The
// freshness thresholds the monitor publishes must follow the cadence its
diff --git a/internal/monitoring/monitor_runtime_settings.go b/internal/monitoring/monitor_runtime_settings.go
index ca59a0a4fc..ab702f4486 100644
--- a/internal/monitoring/monitor_runtime_settings.go
+++ b/internal/monitoring/monitor_runtime_settings.go
@@ -80,6 +80,34 @@ func (m *Monitor) pmgPollingIntervalSetting() time.Duration {
return m.config.PMGPollingInterval
}
+// BasePollInterval returns the clamped base cadence this monitor polls PVE,
+// PBS or PMG at, honouring the runtime polling overrides. For the built-in
+// providers these are the values the fixed-cadence scheduler reads
+// (baseIntervalForInstanceType); a replacement provider or a task already
+// queued at its previous interval can differ until the next planning pass.
+// This accessor skips the poll-provider lookup, so it never takes m.mu.
+// Non-default tenant monitors poll against a detached config copy (#1619), so
+// a saved PBS or PMG interval reaches them only as an override. Code outside
+// the monitor that judges poll freshness must read the cadence here, not from
+// GetConfig. An adaptive scheduler plans its own intervals independently of
+// this base; see PlannedPollInterval. Zero means the cadence is unknown:
+// another instance type, or a monitor without config.
+func (m *Monitor) BasePollInterval(instanceType InstanceType) time.Duration {
+ if m == nil || m.config == nil {
+ return 0
+ }
+ switch instanceType {
+ case InstanceTypePVE:
+ return m.effectivePVEPollingInterval()
+ case InstanceTypePBS:
+ return clampInterval(m.pbsPollingIntervalSetting(), 10*time.Second, time.Hour)
+ case InstanceTypePMG:
+ return clampInterval(m.pmgPollingIntervalSetting(), 10*time.Second, time.Hour)
+ default:
+ return 0
+ }
+}
+
// SetBackupPollingEnabled toggles backup polling on the live monitor.
// Re-enabling clears the per-instance last-poll timestamps so the next
// polling cycle runs an immediate catch-up poll, matching the behavior of
diff --git a/internal/monitoring/resource_stale_thresholds.go b/internal/monitoring/resource_stale_thresholds.go
index 5f32913bc7..92e1468572 100644
--- a/internal/monitoring/resource_stale_thresholds.go
+++ b/internal/monitoring/resource_stale_thresholds.go
@@ -136,13 +136,14 @@ func (m *Monitor) resourceStaleThresholds() map[unifiedresources.DataSource]time
// selects its own intervals and never reads the per-platform overrides, so
// they must not move freshness there either.
func (m *Monitor) resourcePollIntervals() resourcePollIntervals {
- intervals := resourcePollIntervalsForConfig(m.config)
if m.scheduler != nil {
- return intervals
+ return resourcePollIntervalsForConfig(m.config)
+ }
+ return resourcePollIntervals{
+ pve: m.BasePollInterval(InstanceTypePVE),
+ pbs: m.BasePollInterval(InstanceTypePBS),
+ pmg: m.BasePollInterval(InstanceTypePMG),
}
- intervals.pbs = clampInterval(m.pbsPollingIntervalSetting(), 10*time.Second, time.Hour)
- intervals.pmg = clampInterval(m.pmgPollingIntervalSetting(), 10*time.Second, time.Hour)
- return intervals
}
func (m *Monitor) pveNodeOfflineGracePeriod() time.Duration {