Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Judge connection freshness by the monitor's live poll cadence
rcourtman912d7ceb51fb
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 6a8d9af826..ae067a9cff 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9237,6 +9237,15 @@ registration, enrolment, install, update, removal and report identity areunchanged; the per-image memo lives in the registry checker and is prunedeach collection cycle to the images in use.+### Connection freshness cadence only++`internal/api/connections_alerts.go` changed only so PVE, PBS and PMG+connection rows scale their stale cutoff by the monitor's base poll cadence+(`Monitor.BasePollInterval`, which honours runtime polling overrides) rather+than by a non-default org's detached config copy. Agent connection rows keep+their own heartbeat cutoff. Agent registration, enrolment, install, update,+removal and report identity are unchanged.+### Host snapshots report a linked agent's own heartbeat`internal/monitoring/monitor.go` changed only so a host produced from the readdiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 0ec4e56316..53b2a60b58 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -9997,6 +9997,25 @@ metadata, but API freshness and `lastSeenAt` projections use the server-authoredreceipt time supplied by monitoring. API consumers must not substitute theagent clock for disconnect detection or apply a second, shorter genericConnections staleness window.+PVE, PBS and PMG connection rows scale their active-to-stale cutoff by the+cadence the org's monitor polls at.+`buildAggregatorInputsWithRuntimeSources` reads `Monitor.BasePollInterval`+(clamped like the scheduler, so an out-of-range configured interval is judged+by the cadence actually polled) whenever a monitor is present, and falls back+to the request config only when there is no monitor. A non-default org's request config is its monitor's+detached copy (#1619), which a system-settings save does not update; saved PBS+and PMG intervals reach that monitor only as runtime overrides. Reading the+copy left Settings > Infrastructure rows stale for part of every cycle after+an interval was raised, and slow to go stale after one was lowered.+Adaptive polling still scales by the larger of that base and+`PlannedPollInterval`, a conservative floor that can exceed the adaptive+cadence. The connections list, runtime inventory sources and+diagnostics all share this input builder. The system settings GET is not+affected: the router serves `/api/config/system` from `SystemSettingsHandler`,+which reads the base config and saved settings, not a tenant copy.+`TestContract_ConnectionFreshnessFollowsSavedPollingIntervalsInTenantMonitors`+in `internal/api/contract_test.go` drives a real settings save into a tenant+monitor built from a detached copy and checks both directions.Availability target writes now use `observationLocationIds` as the canonicalbounded set. Values are source-owned IDs (`pulse:local` or`agent:<agent-id>`), are normalized and deduplicated by the server, and everydiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 5bb69533b2..58f47f21be 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -3610,6 +3610,30 @@ tighten it only when a freshly shortened interval justifies an earlier run.Without this, a sixty-second availability target polls at the ten-secondtick cadence whenever adaptive polling is off, which is the default.+### Base poll cadence is read through the monitor, not its config copy++`Monitor.BasePollInterval` (`internal/monitoring/monitor_runtime_settings.go`)+is the exported reader for the clamped base cadence the monitor polls PVE,+PBS and PMG at. For the built-in poll providers it returns the same values as+the scheduler's `baseIntervalForInstanceType`, including PBS and PMG runtime+polling overrides, but skips the poll-provider lookup, so it never takes+`m.mu`. A replacement provider's own `BaseInterval`, a provider's fixed+instance interval, and a task already queued at its previous interval can+differ from it until the next planning pass. A monitor without config reports+zero where the scheduler falls back to ten seconds.+Non-default tenant monitors poll against a detached config copy (#1619), so a+saved PBS or PMG interval reaches them only as one of those overrides, and+`GetConfig` keeps the old value until that monitor is rebuilt (a restart, or a+full reload such as a PVE interval save). Code outside the monitor that+judges poll freshness, starting with the connections list, must read the cadence+through `BasePollInterval`, not from the config. It reports the same base under+adaptive scheduling, where it is only a floor: the adaptive scheduler plans+each instance's interval independently of the per-platform intervals, and+`PlannedPollInterval` reports that plan. The fixed-cadence branch of+`Monitor.resourcePollIntervals` reads the same accessor. Regression coverage:+`TestMonitorBasePollIntervalReportsSchedulerBaseCadence` in+`internal/monitoring/canonical_guardrails_test.go`.+### Host report admission does not wedge on stale removal blocksHost-agent report admission consults removal blocks across the durablediff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 34d4e0d1c7..f1bbc4f68a 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -6629,6 +6629,16 @@ to it.Both are read-path changes to source freshness and presentation; no storage,retention, backup or recovery path is added or moved.+### Connection freshness cadence opens no storage or recovery path++`internal/api/connections_alerts.go` now scales the connections list's+active-to-stale cutoff by the monitor's base poll cadence+(`Monitor.BasePollInterval`), which honours saved PBS and PMG intervals that+reach a non-default org's monitor only as runtime overrides. This is a+read-path change to connection freshness. Backup polling, PBS backup and+datastore collection, retention and recovery are unchanged, and nothing new+is persisted.+### Webhook configuration never persists the API mask`internal/api/alerting/notifications.go` resolves each `***REDACTED***`diff --git a/internal/api/connections_aggregator.go b/internal/api/connections_aggregator.goindex dd07879e2c..eb32d90d67 100644--- a/internal/api/connections_aggregator.go+++ b/internal/api/connections_aggregator.go@@ -133,7 +133,9 @@ type aggregatorInputs struct {pbsReportedNodeNames map[string]string// Configured poll cadences, used to scale the active→stale cutoff so slow- // cadences (e.g. 5 minutes) don't read as permanently stale. Zero means+ // cadences (e.g. 5 minutes) don't read as permanently stale. With a+ // monitor present they are its base cadences (Monitor.BasePollInterval,+ // which honours runtime polling overrides), not the config copy. Zero means// unknown and falls back to the connectionStaleThreshold floor. VMware and// TrueNAS cadences come from their summaries/instances instead.pvePollingInterval time.Durationdiff --git a/internal/api/connections_alerts.go b/internal/api/connections_alerts.goindex 7c2229a48e..0a8dbdbf41 100644--- a/internal/api/connections_alerts.go+++ b/internal/api/connections_alerts.go@@ -66,6 +66,19 @@ func buildAggregatorInputsWithRuntimeSources(inputs.instanceHealth = instanceHealthByKey(monitor.SchedulerHealth())inputs.availabilityStatuses = monitor.AvailabilityStatusSnapshot()inputs.pbsReportedNodeNames = pbsReportedNodeNamesByInstance(monitor.PBSInstancesSnapshot())+ // A non-default org's cfg is its monitor's detached config copy+ // (#1619), which a settings save does not update: saved PBS and PMG+ // intervals reach the monitor only as runtime overrides. Scale the+ // stale cutoff by the cadence the monitor actually polls at.+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePVE); interval > 0 {+ inputs.pvePollingInterval = interval+ }+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePBS); interval > 0 {+ inputs.pbsPollingInterval = interval+ }+ if interval := monitor.BasePollInterval(monitoring.InstanceTypePMG); interval > 0 {+ inputs.pmgPollingInterval = interval+ }inputs.plannedPollIntervals = plannedPollIntervalsForConfig(monitor, cfg)} else {inputs.hosts = []models.Host{}diff --git a/internal/api/contract_test.go b/internal/api/contract_test.goindex 5f9d67b9f6..3f6c1b18ab 100644--- a/internal/api/contract_test.go+++ b/internal/api/contract_test.go@@ -3581,6 +3581,97 @@ func TestContract_GenerateStyledMockSeries_UsesTimestampBasedCurve(t *testing.T)}}+// Non-default organisations poll against a detached config copy (#1619). A+// system-settings save updates the server's base config and pushes saved PBS+// and PMG intervals into each live tenant monitor as runtime overrides, but+// the copy that configapi hands that org's connections list (m.GetConfig())+// keeps the old values. Connection freshness must scale by the cadence the+// monitor polls at, or Settings > Infrastructure rows read stale for part of+// every slower cycle.+func TestContract_ConnectionFreshnessFollowsSavedPollingIntervalsInTenantMonitors(t *testing.T) {+ setMockModeForTest(t, false)++ cases := []struct {+ name string+ configured time.Duration+ saved time.Duration+ wantState ConnectionState+ }{+ {+ name: "raised interval keeps an on-schedule poll active",+ configured: time.Minute,+ saved: 5 * time.Minute,+ wantState: ConnectionStateActive,+ },+ {+ name: "lowered interval judges the same poll stale",+ configured: 5 * time.Minute,+ saved: time.Minute,+ wantState: ConnectionStateStale,+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ tempDir := t.TempDir()+ base := &config.Config{+ DataPath: tempDir,+ ConfigPath: tempDir,+ PVEPollingInterval: 10 * time.Second,+ PBSPollingInterval: tc.configured,+ PMGPollingInterval: tc.configured,+ }+ tenant, err := monitoring.New(base.DeepCopy())+ if err != nil {+ t.Fatalf("new tenant monitor: %v", err)+ }+ t.Cleanup(tenant.Stop)++ tokenVal := "tenant-polling-token"+ base.APITokens = []config.APITokenRecord{+ {ID: "token1", Hash: authpkg.HashAPIToken(tokenVal), Name: "Test Token"},+ }+ handler := newTestSystemSettingsHandler(base, config.NewConfigPersistence(tempDir), tenant, func() {}, func() error { return nil })+ seconds := int(tc.saved / time.Second)+ body, _ := json.Marshal(map[string]any{"pbsPollingInterval": seconds, "pmgPollingInterval": seconds})+ req := httptest.NewRequest(http.MethodPost, "/api/system-settings", bytes.NewReader(body))+ req.Header.Set("X-API-Token", tokenVal)+ rec := httptest.NewRecorder()+ handler.HandleUpdateSystemSettings(rec, req)+ if rec.Code != http.StatusOK {+ t.Fatalf("save status = %d, body: %s", rec.Code, rec.Body.String())+ }++ cfg := tenant.GetConfig()+ if base.PBSPollingInterval != tc.saved || cfg.PBSPollingInterval != tc.configured {+ t.Fatalf("PBS interval base = %v, tenant copy = %v; want the save to reach only the base (%v) and the copy to keep %v",+ base.PBSPollingInterval, cfg.PBSPollingInterval, tc.saved, tc.configured)+ }++ now := time.Now()+ lastSuccess := now.Add(-4 * time.Minute)+ inputs := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, tenant, aggregatorRuntimeSources{orgID: "tenant-a"})+ inputs.now = now+ inputs.pbsInstances = []config.PBSInstance{{Name: "backup", Host: "https://pbs.lab.local:8007"}}+ inputs.pmgInstances = []config.PMGInstance{{Name: "mail", Host: "https://pmg.lab.local:8006"}}+ inputs.instanceHealth = map[string]monitoring.InstanceHealth{+ "pbs::backup": {PollStatus: monitoring.InstancePollStatus{LastSuccess: &lastSuccess}},+ "pmg::mail": {PollStatus: monitoring.InstancePollStatus{LastSuccess: &lastSuccess}},+ }++ states := map[ConnectionType]ConnectionState{}+ for _, conn := range buildConnections(inputs) {+ states[conn.Type] = conn.State+ }+ for _, connType := range []ConnectionType{ConnectionTypePBS, ConnectionTypePMG} {+ if got := states[connType]; got != tc.wantState {+ t.Errorf("%s state = %q, want %q for a 4m-old poll after saving %v (tenant copy still %v)",+ connType, got, tc.wantState, tc.saved, tc.configured)+ }+ }+ })+ }+}+func TestContract_PlatformMockToggleRebindsRuntimeConnectionsAndResources(t *testing.T) {setMockModeForTest(t, false)diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.goindex 4472c7f966..7d409f0fc0 100644--- a/internal/monitoring/canonical_guardrails_test.go+++ b/internal/monitoring/canonical_guardrails_test.go@@ -2888,6 +2888,83 @@ func TestResourceStaleThresholdsPreserveDefaultFloors(t *testing.T) {}}+// Consumers outside the monitor, such as the connections list, judge poll+// freshness by Monitor.BasePollInterval. It must report the base cadence the+// scheduler polls each platform at, including runtime polling overrides on a+// detached tenant config copy (#1619), and must not take m.mu, which the+// scheduler's poll-provider lookup takes.+func TestMonitorBasePollIntervalReportsSchedulerBaseCadence(t *testing.T) {+ newTenantMonitor := func(override time.Duration) *Monitor {+ base := &config.Config{+ PVEPollingInterval: 30 * time.Second,+ PBSPollingInterval: time.Minute,+ PMGPollingInterval: time.Minute,+ }+ monitor := &Monitor{config: base.DeepCopy()}+ monitor.SetPBSPollingInterval(override)+ monitor.SetPMGPollingInterval(override)+ return monitor+ }++ t.Run("runtime overrides on a detached copy, clamped like the scheduler", func(t *testing.T) {+ for override, want := range map[time.Duration]time.Duration{+ 5 * time.Second: 10 * time.Second,+ 90 * time.Second: 90 * time.Second,+ 5 * time.Minute: 5 * time.Minute,+ 2 * time.Hour: time.Hour,+ } {+ monitor := newTenantMonitor(override)+ for _, instanceType := range []InstanceType{InstanceTypePBS, InstanceTypePMG} {+ if got := monitor.BasePollInterval(instanceType); got != want {+ t.Errorf("%s base interval = %v for override %v, want %v", instanceType, got, override, want)+ }+ }+ for _, instanceType := range []InstanceType{InstanceTypePVE, InstanceTypePBS, InstanceTypePMG} {+ if got, scheduled := monitor.BasePollInterval(instanceType), monitor.baseIntervalForInstanceType(instanceType); got != scheduled {+ t.Errorf("%s base interval = %v for override %v, want the scheduler's %v", instanceType, got, override, scheduled)+ }+ }+ }+ })++ t.Run("adaptive scheduling keeps the same base", func(t *testing.T) {+ monitor := newTenantMonitor(5 * time.Minute)+ monitor.scheduler = NewAdaptiveScheduler(SchedulerConfig{}, nil, nil, nil)+ if got := monitor.BasePollInterval(InstanceTypePBS); got != 5*time.Minute {+ t.Fatalf("PBS base interval = %v under adaptive scheduling, want the saved %v", got, 5*time.Minute)+ }+ })++ t.Run("does not take the monitor lock", func(t *testing.T) {+ monitor := newTenantMonitor(5 * time.Minute)+ monitor.mu.Lock()+ defer monitor.mu.Unlock()+ done := make(chan time.Duration, 1)+ go func() { done <- monitor.BasePollInterval(InstanceTypePBS) }()+ select {+ case got := <-done:+ if got != 5*time.Minute {+ t.Fatalf("PBS base interval = %v, want %v", got, 5*time.Minute)+ }+ case <-time.After(2 * time.Second):+ t.Fatal("BasePollInterval blocked while m.mu was held")+ }+ })++ t.Run("unknown cadence reads zero", func(t *testing.T) {+ var nilMonitor *Monitor+ for name, got := range map[string]time.Duration{+ "nil monitor": nilMonitor.BasePollInterval(InstanceTypePBS),+ "monitor no config": (&Monitor{}).BasePollInterval(InstanceTypePVE),+ "availability": newTenantMonitor(time.Minute).BasePollInterval(InstanceTypeAvailability),+ } {+ if got != 0 {+ t.Errorf("%s base interval = %v, want 0", name, got)+ }+ }+ })+}+// Non-default tenant monitors poll against a detached config copy (#1619), so// a saved PBS or PMG interval reaches them only as a runtime override. The// freshness thresholds the monitor publishes must follow the cadence itsdiff --git a/internal/monitoring/monitor_runtime_settings.go b/internal/monitoring/monitor_runtime_settings.goindex ca59a0a4fc..ab702f4486 100644--- a/internal/monitoring/monitor_runtime_settings.go+++ b/internal/monitoring/monitor_runtime_settings.go@@ -80,6 +80,34 @@ func (m *Monitor) pmgPollingIntervalSetting() time.Duration {return m.config.PMGPollingInterval}+// BasePollInterval returns the clamped base cadence this monitor polls PVE,+// PBS or PMG at, honouring the runtime polling overrides. For the built-in+// providers these are the values the fixed-cadence scheduler reads+// (baseIntervalForInstanceType); a replacement provider or a task already+// queued at its previous interval can differ until the next planning pass.+// This accessor skips the poll-provider lookup, so it never takes m.mu.+// Non-default tenant monitors poll against a detached config copy (#1619), so+// a saved PBS or PMG interval reaches them only as an override. Code outside+// the monitor that judges poll freshness must read the cadence here, not from+// GetConfig. An adaptive scheduler plans its own intervals independently of+// this base; see PlannedPollInterval. Zero means the cadence is unknown:+// another instance type, or a monitor without config.+func (m *Monitor) BasePollInterval(instanceType InstanceType) time.Duration {+ if m == nil || m.config == nil {+ return 0+ }+ switch instanceType {+ case InstanceTypePVE:+ return m.effectivePVEPollingInterval()+ case InstanceTypePBS:+ return clampInterval(m.pbsPollingIntervalSetting(), 10*time.Second, time.Hour)+ case InstanceTypePMG:+ return clampInterval(m.pmgPollingIntervalSetting(), 10*time.Second, time.Hour)+ default:+ return 0+ }+}+// SetBackupPollingEnabled toggles backup polling on the live monitor.// Re-enabling clears the per-instance last-poll timestamps so the next// polling cycle runs an immediate catch-up poll, matching the behavior ofdiff --git a/internal/monitoring/resource_stale_thresholds.go b/internal/monitoring/resource_stale_thresholds.goindex 5f32913bc7..92e1468572 100644--- a/internal/monitoring/resource_stale_thresholds.go+++ b/internal/monitoring/resource_stale_thresholds.go@@ -136,13 +136,14 @@ func (m *Monitor) resourceStaleThresholds() map[unifiedresources.DataSource]time// selects its own intervals and never reads the per-platform overrides, so// they must not move freshness there either.func (m *Monitor) resourcePollIntervals() resourcePollIntervals {- intervals := resourcePollIntervalsForConfig(m.config)if m.scheduler != nil {- return intervals+ return resourcePollIntervalsForConfig(m.config)+ }+ return resourcePollIntervals{+ pve: m.BasePollInterval(InstanceTypePVE),+ pbs: m.BasePollInterval(InstanceTypePBS),+ pmg: m.BasePollInterval(InstanceTypePMG),}- intervals.pbs = clampInterval(m.pbsPollingIntervalSetting(), 10*time.Second, time.Hour)- intervals.pmg = clampInterval(m.pmgPollingIntervalSetting(), 10*time.Second, time.Hour)- return intervals}func (m *Monitor) pveNodeOfflineGracePeriod() time.Duration {