Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Judge adaptive-polling freshness by the scheduler's cadence

rcourtmandc93d0adea49
diff --git a/docs/monitoring/ADAPTIVE_POLLING.md b/docs/monitoring/ADAPTIVE_POLLING.md
index a7036ce187..b0f7604b55 100644
--- a/docs/monitoring/ADAPTIVE_POLLING.md
+++ b/docs/monitoring/ADAPTIVE_POLLING.md
@@ -47,13 +47,30 @@ There is currently no dedicated UI for adaptive polling. Use environment variabl
| Variable | Default | Description |
| :--- | :--- | :--- |
| `ADAPTIVE_POLLING_ENABLED` | `false` | Enable/disable. |
-| `ADAPTIVE_POLLING_BASE_INTERVAL` | `10s` | Healthy poll rate. |
-| `ADAPTIVE_POLLING_MIN_INTERVAL` | `5s` | Active/busy rate. |
-| `ADAPTIVE_POLLING_MAX_INTERVAL` | `5m` | Idle/backoff rate. |
+| `ADAPTIVE_POLLING_BASE_INTERVAL` | `10s` | Starting interval before an instance has a schedule. |
+| `ADAPTIVE_POLLING_MIN_INTERVAL` | `5s` | Shortest interval, approached as data goes unrefreshed. |
+| `ADAPTIVE_POLLING_MAX_INTERVAL` | `5m` | Longest interval. Healthy instances approach it. |
### system.json
You can also set `adaptivePollingEnabled` (and related interval fields) in `system.json` and restart Pulse.
+### What changes when it is on
+Adaptive polling replaces the Proxmox VE, PBS and PMG polling intervals
+(Settings → General → Monitoring cadence, `PBS_POLLING_INTERVAL`,
+`PMG_POLLING_INTERVAL`) for those platforms. Each successful poll resets an
+instance's staleness score, so the scheduler stretches a healthy instance's
+interval toward `ADAPTIVE_POLLING_MAX_INTERVAL`. With the defaults, healthy
+instances are polled every few minutes, not every 10 seconds.
+
+Freshness follows that cadence. A Proxmox VE, PBS or PMG resource is shown as
+stale only when its last sighting from a poll is older than twice the maximum
+interval (10 minutes with the defaults). A Proxmox node that stops answering can
+keep showing online for the same window after it was last seen online, although
+a cluster's own report that a member is offline applies immediately. Offline
+nodes and lost data therefore take longer to show than with fixed polling. For a
+lighter but predictable load, raise the polling intervals instead: freshness and
+offline detection then scale with the interval you choose.
+
## 📊 Metrics
The separate metrics listener defaults to port 9091; `METRICS_PORT` can override
it. Use an existing authorised scrape if available, not a newly exposed port for
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index df749fdb7f..25c9a7a177 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -1743,9 +1743,11 @@ guest-agent reachability: fresh or never-healthy VMs with an enabled but
unavailable guest agent stay `not-running`, while only VMs with recent healthy
guest-agent evidence may become `expected-unreachable`.
Monitoring owns source freshness cadence for Proxmox, PBS, and PMG resources:
-the stale threshold is derived from the configured polling interval with a
-minimum floor, so API-facing resource status must not degrade merely because a
-healthy source is between normal poll cycles.
+the stale threshold is derived from the cadence the monitor actually polls at
+(the configured polling interval, or the adaptive scheduler's maximum interval
+when adaptive polling is enabled) with a minimum floor, so API-facing resource
+status must not degrade merely because a healthy source is between normal poll
+cycles.
Each unified view carries the thresholds its listed registry generation was
judged by, read with the listing from the same generation: the resource
store's configured ones, or none (the defaults) for a view built from mock
@@ -2027,9 +2029,11 @@ VMID)`: it survives node migration, separates QEMU from LXC, and prevents
duplicate configured cluster identities from sharing a concurrent baseline.
Idle and partial samples still refresh tracker liveness.
Proxmox row liveness uses the same cadence-derived threshold as source
-freshness (`max(2 * configured poll interval, 60s)`). Node offline grace and
-guest preservation must not expire between healthy 60- or 90-second polls, and
-must not use a separate fixed 60-second timer.
+freshness (`max(2 * poll cadence, 60s)`, the cadence being the configured poll
+interval or, under adaptive polling, the scheduler's maximum interval). Node
+offline grace and guest preservation must not expire between healthy 60- or
+90-second polls, or between adaptive polls, and must not use a separate fixed
+60-second timer.
Tenant monitor enumeration is monitoring-owned runtime topology, not a
reporting source of truth. `MultiTenantMonitor.ListOrganizationIDs` may expose
persisted organization IDs to API-owned background workers, but it must not
@@ -2764,14 +2768,40 @@ service-history reads plus denial/recovery without fabricated samples.
overrides. Under fixed-cadence scheduling, `Monitor.resourceStaleThresholds`
reads those overrides through the same clamped setting readers as the
scheduler, so a saved interval moves the monitor's resource freshness and
- polling together. An adaptive scheduler selects its own intervals and
- ignores those overrides, so freshness there keeps deriving from the
- configured per-platform intervals. A PVE interval save reloads every
- monitor from saved config instead. `ResourceStaleThresholdsForConfig` stays
- the config-only derivation for callers without a live monitor, such as
- adapter construction. Regression coverage:
+ polling together. A PVE interval save reloads every monitor from saved
+ config instead. `ResourceStaleThresholdsForConfig` stays the config-only
+ derivation for callers without a live monitor, such as adapter
+ construction. Regression coverage:
`TestMonitorResourceStaleThresholdsFollowRuntimePollingOverrides` in
`internal/monitoring/canonical_guardrails_test.go`.
+ An adaptive scheduler (`ADAPTIVE_POLLING_ENABLED` or `system.json`; off by
+ default, no UI) selects its own intervals and never reads the per-platform
+ intervals or those overrides. A healthy instance's staleness score is near
+ zero after each success, so its cadence stretches toward the scheduler's
+ maximum interval (5 minutes by default; a healthy instance polls a few
+ minutes apart). Selected intervals never exceed it; a poll can still run
+ later than its slot while workers are busy; the factor of two gives
+ headroom for that. Under adaptive scheduling
+ `Monitor.resourcePollIntervals` therefore derives PVE, PBS and PMG
+ freshness from `AdaptiveScheduler.MaxInterval()`, and
+ `ResourceStaleThresholdsForConfig` derives it from the
+ `AdaptivePolling*Interval` bounds normalized by `normalizedSchedulerConfig`,
+ the normalization the scheduler itself applies. Deriving it from the
+ per-platform intervals (1 minute for PBS and PMG, 10 seconds for PVE) left
+ healthy rows stale for much of every cycle. The Proxmox threshold is also
+ the window the node grace helpers (`determineNodeIDAndStatus`,
+ `preserveOrExpireNodes`) hold a node online after it was last seen online,
+ and the node temperature carry window derives from the same cadence with
+ its five-minute floor. With the defaults that hold is 10 minutes, observed
+ at the next poll after it lapses: a longer detection delay than fixed
+ polling, which is the price of polling a healthy fleet every few minutes.
+ A valid `/cluster/status` membership report that a member is offline is
+ authoritative (`reconcilePVENodeInventory`) and still applies immediately.
+ Regression coverage:
+ `TestMonitorResourceFreshnessFollowsAdaptiveCadence`,
+ `TestResourceStaleThresholdsForConfigFollowAdaptiveBounds` and
+ `TestAdaptiveNodeOfflineGraceSpansTheAdaptiveCycle` in
+ `internal/monitoring/adaptive_resource_freshness_test.go`.
Periodic out-of-scheduler platform pollers (TrueNAS, VMware) share their
lifecycle and config-resolution scaffold through
`internal/monitoring/platform_poller_shared.go`: `startPollerLoop` owns
@@ -5072,7 +5102,8 @@ reading with that report time. Past the lease the lookup falls back to the
cluster sensor cache, which keeps its own recency check. When every source
returns nothing, `internal/monitoring/monitor_polling_node_helpers.go` may carry
a previous reading, with its original `LastUpdate`, only inside the carry window
-(twice the PVE polling interval, never under five minutes); an older reading is
+(twice the PVE polling cadence, which under adaptive polling is the
+scheduler's maximum interval, never under five minutes); an older reading is
dropped rather than re-presented as current. A carried reading also keeps a
lapsed host agent's lease. Agent readings are stamped with the agent's report
time, so `carriedTemperatureOutlivesAgentLease` drops a carried reading stamped
diff --git a/internal/monitoring/adaptive_resource_freshness_test.go b/internal/monitoring/adaptive_resource_freshness_test.go
new file mode 100644
index 0000000000..87c99343f6
--- /dev/null
+++ b/internal/monitoring/adaptive_resource_freshness_test.go
@@ -0,0 +1,234 @@
+package monitoring
+
+import (
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+// simulateAdaptiveSuccessGaps drives the real adaptive scheduler, interval
+// selector and staleness scoring for one always-healthy instance on a
+// simulated clock, the way the monitor does: a worker runs the queued task
+// once it is due and reschedules it, and a planning pass re-plans every poll
+// tick. It returns the gaps between successive successful polls after warmup.
+// The worker is instantaneous and uncontended, so real gaps can run longer
+// when workers are busy; the factor of two in the threshold is headroom for
+// that, not a bound.
+func simulateAdaptiveSuccessGaps(t *testing.T, cfg SchedulerConfig, instanceType InstanceType, tick, warmup, horizon time.Duration) []time.Duration {
+ t.Helper()
+ const name = "instance-1"
+ tracker := NewStalenessTracker(nil)
+ tracker.SetBounds(cfg.BaseInterval, cfg.MaxInterval)
+ scheduler := NewAdaptiveScheduler(cfg, tracker, nil, nil)
+
+ start := time.Unix(1_800_000_000, 0)
+ simNow := start
+ var lastSuccess time.Time
+ plan := func(desc InstanceDescriptor) ScheduledTask {
+ // StalenessTracker scores against the wall clock, so stamp the
+ // simulated age of the last success just before planning.
+ if !lastSuccess.IsZero() {
+ tracker.setSnapshot(FreshnessSnapshot{
+ InstanceType: instanceType,
+ Instance: name,
+ LastSuccess: time.Now().Add(-simNow.Sub(lastSuccess)),
+ })
+ }
+ tasks := scheduler.BuildPlan(simNow, []InstanceDescriptor{desc}, 0)
+ if len(tasks) != 1 {
+ t.Fatalf("BuildPlan returned %d tasks, want 1", len(tasks))
+ }
+ return tasks[0]
+ }
+ planningPass := func() ScheduledTask {
+ desc := InstanceDescriptor{Name: name, Type: instanceType, LastSuccess: lastSuccess}
+ if last, ok := scheduler.LastScheduled(instanceType, name); ok {
+ desc.LastScheduled = last.NextRun
+ desc.LastInterval = last.Interval
+ }
+ return plan(desc)
+ }
+
+ queued := planningPass()
+ var gaps []time.Duration
+ for step := time.Duration(0); step <= horizon; step += time.Second {
+ simNow = start.Add(step)
+ if !simNow.Before(queued.NextRun) {
+ if !lastSuccess.IsZero() && step >= warmup {
+ gaps = append(gaps, simNow.Sub(lastSuccess))
+ }
+ lastSuccess = simNow
+ queued = plan(InstanceDescriptor{
+ Name: name,
+ Type: instanceType,
+ LastInterval: queued.Interval,
+ LastScheduled: queued.NextRun,
+ LastSuccess: lastSuccess,
+ })
+ }
+ if step > 0 && step%tick == 0 {
+ queued = planningPass()
+ }
+ }
+ if len(gaps) == 0 {
+ t.Fatalf("%s: no polls after warmup", instanceType)
+ }
+ return gaps
+}
+
+func longestGap(gaps []time.Duration) time.Duration {
+ var longest time.Duration
+ for _, gap := range gaps {
+ longest = max(longest, gap)
+ }
+ return longest
+}
+
+// With adaptive polling enabled, a healthy instance's staleness score is near
+// zero after every success, so the scheduler stretches its cadence toward
+// AdaptivePollingMaxInterval and never reads the per-platform intervals.
+// Freshness, node offline grace and the temperature carry window must cover
+// that cadence, or healthy Proxmox, PBS and PMG rows read stale (and nodes
+// lose their grace) for most of every cycle.
+func TestMonitorResourceFreshnessFollowsAdaptiveCadence(t *testing.T) {
+ sources := map[InstanceType]unifiedresources.DataSource{
+ InstanceTypePVE: unifiedresources.SourceProxmox,
+ InstanceTypePBS: unifiedresources.SourcePBS,
+ InstanceTypePMG: unifiedresources.SourcePMG,
+ }
+ for _, bounds := range []SchedulerConfig{
+ {BaseInterval: 10 * time.Second, MinInterval: 5 * time.Second, MaxInterval: 5 * time.Minute},
+ {BaseInterval: 30 * time.Second, MinInterval: 10 * time.Second, MaxInterval: 15 * time.Minute},
+ } {
+ monitor := &Monitor{
+ config: &config.Config{
+ PVEPollingInterval: 10 * time.Second,
+ PBSPollingInterval: time.Minute,
+ PMGPollingInterval: time.Minute,
+ },
+ scheduler: NewAdaptiveScheduler(bounds, nil, nil, nil),
+ }
+ thresholds := monitor.resourceStaleThresholds()
+ for instanceType, source := range sources {
+ gaps := simulateAdaptiveSuccessGaps(t, bounds, instanceType, 10*time.Second, time.Hour, 8*time.Hour)
+ longest := longestGap(gaps)
+ if longest > bounds.MaxInterval+time.Second {
+ t.Fatalf("%s: healthy polls %v apart exceed the scheduler's %v maximum", instanceType, longest, bounds.MaxInterval)
+ }
+ if got, want := thresholds[source], 2*bounds.MaxInterval; got != want {
+ t.Errorf("max %v: %s threshold = %v, want %v from the adaptive cadence (healthy polls up to %v apart)",
+ bounds.MaxInterval, source, got, want, longest)
+ }
+ if instanceType != InstanceTypePVE {
+ continue
+ }
+ if grace := monitor.pveNodeOfflineGracePeriod(); grace <= longest {
+ t.Errorf("max %v: node offline grace %v expires before the next healthy poll (%v apart)", bounds.MaxInterval, grace, longest)
+ }
+ if carry, want := monitor.nodeTemperatureCarryWindow(), max(2*bounds.MaxInterval, nodeTemperatureCarryFloor); carry != want || carry <= longest {
+ t.Errorf("max %v: temperature carry window = %v, want %v from the adaptive cadence (healthy polls up to %v apart)", bounds.MaxInterval, carry, want, longest)
+ }
+ }
+ }
+}
+
+// A node the API reports offline is only believed once it has been away
+// longer than one healthy adaptive cycle; past twice the cadence the grace
+// lapses as it does under fixed-cadence polling.
+func TestAdaptiveNodeOfflineGraceSpansTheAdaptiveCycle(t *testing.T) {
+ instance := &config.PVEInstance{Name: "site-a"}
+ monitor := &Monitor{
+ config: &config.Config{PVEPollingInterval: 10 * time.Second},
+ scheduler: NewAdaptiveScheduler(SchedulerConfig{BaseInterval: 10 * time.Second, MinInterval: 5 * time.Second, MaxInterval: 5 * time.Minute}, nil, nil, nil),
+ nodeLastOnline: map[string]time.Time{"site-a-pve-a": time.Now().Add(-4 * time.Minute)},
+ }
+ offline := proxmox.Node{Node: "pve-a", Status: "offline"}
+ if _, status := monitor.determineNodeIDAndStatus("site-a", instance, offline); status != "online" {
+ t.Fatalf("node status = %q one adaptive cycle after its last online poll, want online inside the grace", status)
+ }
+ monitor.nodeLastOnline["site-a-pve-a"] = time.Now().Add(-11 * time.Minute)
+ if _, status := monitor.determineNodeIDAndStatus("site-a", instance, offline); status != "offline" {
+ t.Fatalf("node status = %q past twice the adaptive cadence, want offline", status)
+ }
+}
+
+// Callers without a live monitor (adapter construction) derive freshness from
+// config alone. It must name the same cadence the monitor's scheduler will
+// run, including the defaults the scheduler fills in for unset or inconsistent
+// adaptive bounds.
+func TestResourceStaleThresholdsForConfigFollowAdaptiveBounds(t *testing.T) {
+ cases := []struct {
+ name string
+ cfg config.Config
+ want time.Duration
+ }{
+ {
+ name: "adaptive off keeps the per-platform intervals",
+ cfg: config.Config{
+ PVEPollingInterval: 10 * time.Second, PBSPollingInterval: 3 * time.Minute, PMGPollingInterval: 3 * time.Minute,
+ AdaptivePollingMaxInterval: 15 * time.Minute,
+ },
+ },
+ {
+ name: "configured maximum",
+ cfg: config.Config{
+ AdaptivePollingEnabled: true,
+ PBSPollingInterval: 3 * time.Minute,
+ AdaptivePollingMinInterval: 5 * time.Second, AdaptivePollingMaxInterval: 15 * time.Minute,
+ },
+ want: 30 * time.Minute,
+ },
+ {
+ name: "unset bounds use the scheduler default",
+ cfg: config.Config{AdaptivePollingEnabled: true},
+ want: 2 * DefaultSchedulerConfig().MaxInterval,
+ },
+ {
+ name: "maximum below the minimum uses the scheduler default",
+ cfg: config.Config{
+ AdaptivePollingEnabled: true,
+ AdaptivePollingMinInterval: time.Minute, AdaptivePollingMaxInterval: 30 * time.Second,
+ },
+ want: 2 * DefaultSchedulerConfig().MaxInterval,
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ cfg := tc.cfg
+ thresholds := ResourceStaleThresholdsForConfig(&cfg)
+ for _, source := range []unifiedresources.DataSource{unifiedresources.SourceProxmox, unifiedresources.SourcePBS, unifiedresources.SourcePMG} {
+ want := tc.want
+ if !cfg.AdaptivePollingEnabled {
+ want = map[unifiedresources.DataSource]time.Duration{
+ unifiedresources.SourceProxmox: defaultProxmoxResourceStaleThreshold,
+ unifiedresources.SourcePBS: 6 * time.Minute,
+ unifiedresources.SourcePMG: 6 * time.Minute,
+ }[source]
+ }
+ if got := thresholds[source]; got != want {
+ t.Errorf("%s threshold = %v, want %v", source, got, want)
+ }
+ }
+ if !cfg.AdaptivePollingEnabled {
+ return
+ }
+ scheduler := NewAdaptiveScheduler(SchedulerConfig{
+ BaseInterval: cfg.AdaptivePollingBaseInterval,
+ MinInterval: cfg.AdaptivePollingMinInterval,
+ MaxInterval: cfg.AdaptivePollingMaxInterval,
+ }, nil, nil, nil)
+ monitor := &Monitor{config: &cfg, scheduler: scheduler}
+ for source, want := range thresholds {
+ if source != unifiedresources.SourceProxmox && source != unifiedresources.SourcePBS && source != unifiedresources.SourcePMG {
+ continue
+ }
+ if got := monitor.resourceStaleThresholds()[source]; got != want {
+ t.Errorf("%s: monitor threshold %v differs from the config-only %v", source, got, want)
+ }
+ }
+ })
+ }
+}
diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go
index 7d409f0fc0..d1ee720c87 100644
--- a/internal/monitoring/canonical_guardrails_test.go
+++ b/internal/monitoring/canonical_guardrails_test.go
@@ -3078,22 +3078,26 @@ func TestMonitorResourceStaleThresholdsFollowRuntimePollingOverrides(t *testing.
})
// The adaptive scheduler picks intervals from its own bounds and never
- // reads the per-platform overrides, so a save must not move freshness
- // there; following adaptive cadence is a separate derivation.
+ // reads the per-platform intervals or overrides, so a save must not move
+ // freshness there: it follows the scheduler's maximum interval, which the
+ // 5-minute per-platform interval below would not give (10 minutes).
+ // TestMonitorResourceFreshnessFollowsAdaptiveCadence covers the cadence
+ // itself.
t.Run("adaptive scheduling ignores per-platform overrides", func(t *testing.T) {
base := &config.Config{PBSPollingInterval: 5 * time.Minute, PMGPollingInterval: 5 * time.Minute}
monitor := &Monitor{
config: base.DeepCopy(),
- scheduler: NewAdaptiveScheduler(SchedulerConfig{}, nil, nil, nil),
+ scheduler: NewAdaptiveScheduler(SchedulerConfig{MaxInterval: 15 * time.Minute}, nil, nil, nil),
}
+ const want = 30 * time.Minute
before := monitor.resourceStaleThresholds()
monitor.SetPBSPollingInterval(time.Minute)
monitor.SetPMGPollingInterval(time.Minute)
after := monitor.resourceStaleThresholds()
- for _, source := range []unifiedresources.DataSource{unifiedresources.SourcePBS, unifiedresources.SourcePMG} {
- if before[source] != 10*time.Minute || after[source] != before[source] {
- t.Errorf("%s threshold = %v -> %v across an override the adaptive scheduler ignores, want %v throughout",
- source, before[source], after[source], 10*time.Minute)
+ for _, source := range []unifiedresources.DataSource{unifiedresources.SourcePBS, unifiedresources.SourcePMG, unifiedresources.SourceProxmox} {
+ if before[source] != want || after[source] != want {
+ t.Errorf("%s threshold = %v -> %v across an override the adaptive scheduler ignores, want %v from its 15m maximum throughout",
+ source, before[source], after[source], want)
}
}
})
diff --git a/internal/monitoring/monitor_polling_node_helpers.go b/internal/monitoring/monitor_polling_node_helpers.go
index dd5e9abba9..4a012cf4b0 100644
--- a/internal/monitoring/monitor_polling_node_helpers.go
+++ b/internal/monitoring/monitor_polling_node_helpers.go
@@ -136,9 +136,11 @@ const nodeTemperatureCarryFloor = 5 * time.Minute
// nodeTemperatureCarryWindow bounds how old a carried-over node temperature may
// be. Beyond it the reading no longer describes the node, so it stops being
-// presented as available.
+// presented as available. It follows the PVE cadence the resource freshness
+// and node offline grace use, so a node preserved inside that grace keeps a
+// reading from its last healthy poll.
func (m *Monitor) nodeTemperatureCarryWindow() time.Duration {
- return resourceStaleThresholdForPollInterval(effectivePVEPollingIntervalForConfig(m.config), nodeTemperatureCarryFloor)
+ return resourceStaleThresholdForPollInterval(m.resourcePollIntervals().pve, nodeTemperatureCarryFloor)
}
func (m *Monitor) collectNodeTemperatureData(
diff --git a/internal/monitoring/resource_stale_thresholds.go b/internal/monitoring/resource_stale_thresholds.go
index 92e1468572..b0c2c8af5e 100644
--- a/internal/monitoring/resource_stale_thresholds.go
+++ b/internal/monitoring/resource_stale_thresholds.go
@@ -21,7 +21,25 @@ type resourcePollIntervals struct {
pmg time.Duration
}
+// adaptiveResourcePollIntervals are the cadences freshness is derived from when
+// an adaptive scheduler plans polling. It selects its own intervals and never
+// reads the per-platform ones; a healthy instance's staleness score is near
+// zero after every success, so its cadence stretches toward the scheduler's
+// maximum interval, which bounds it. Freshness must cover that gap, or
+// healthy Proxmox, PBS and PMG rows read stale, and Proxmox nodes lose their
+// offline grace, for much of every cycle.
+func adaptiveResourcePollIntervals(maxInterval time.Duration) resourcePollIntervals {
+ return resourcePollIntervals{pve: maxInterval, pbs: maxInterval, pmg: maxInterval}
+}
+
func resourcePollIntervalsForConfig(cfg *config.Config) resourcePollIntervals {
+ if cfg != nil && cfg.AdaptivePollingEnabled {
+ return adaptiveResourcePollIntervals(normalizedSchedulerConfig(SchedulerConfig{
+ BaseInterval: cfg.AdaptivePollingBaseInterval,
+ MinInterval: cfg.AdaptivePollingMinInterval,
+ MaxInterval: cfg.AdaptivePollingMaxInterval,
+ }).MaxInterval)
+ }
return resourcePollIntervals{
pve: effectivePVEPollingIntervalForConfig(cfg),
pbs: effectivePlatformPollingIntervalForConfig(cfg, "pbs"),
@@ -30,10 +48,12 @@ func resourcePollIntervalsForConfig(cfg *config.Config) resourcePollIntervals {
}
// ResourceStaleThresholdsForConfig derives canonical resource freshness from
-// configured polling cadence. A source should not be considered stale until it
-// has missed at least one expected poll cycle plus the normal interval. A
-// running monitor judges by Monitor.resourceStaleThresholds instead, which
-// honours the runtime polling overrides its scheduler reads.
+// configured polling cadence: the per-platform intervals, or the adaptive
+// scheduler's maximum interval when adaptive polling is enabled. A source
+// should not be considered stale until it has missed at least one expected
+// poll cycle plus the normal interval. A running monitor judges by
+// Monitor.resourceStaleThresholds instead, which honours the runtime polling
+// overrides its scheduler reads.
func ResourceStaleThresholdsForConfig(cfg *config.Config) map[unifiedresources.DataSource]time.Duration {
return resourceStaleThresholdsForConfig(cfg, mock.IsMockEnabled(), mock.SupplementalRefreshInterval)
}
@@ -126,18 +146,19 @@ func (m *Monitor) resourceStaleThresholds() map[unifiedresources.DataSource]time
)
}
-// resourcePollIntervals reads the per-platform base cadences, honouring the
-// runtime overrides a fixed-cadence scheduler polls at. Non-default tenant
-// monitors poll against a detached config copy (#1619), so the settings API
-// pushes saved PBS and PMG intervals into them as runtime overrides; freshness
-// derived from m.config alone would judge those sources by an interval nothing
-// polls at. A PVE interval change reloads every monitor from saved config
-// instead, so the config value is already the live one. An adaptive scheduler
-// selects its own intervals and never reads the per-platform overrides, so
-// they must not move freshness there either.
+// resourcePollIntervals reads the cadences the monitor's scheduler polls at.
+// A fixed-cadence scheduler polls each platform at its base interval,
+// honouring the runtime overrides: non-default tenant monitors poll against a
+// detached config copy (#1619), so the settings API pushes saved PBS and PMG
+// intervals into them as overrides, and freshness derived from m.config alone
+// would judge those sources by an interval nothing polls at. A PVE interval
+// change reloads every monitor from saved config instead, so the config value
+// is already the live one. An adaptive scheduler never reads the per-platform
+// intervals or their overrides, so they cannot move freshness there; it is
+// judged by the longest interval that scheduler selects.
func (m *Monitor) resourcePollIntervals() resourcePollIntervals {
if m.scheduler != nil {
- return resourcePollIntervalsForConfig(m.config)
+ return adaptiveResourcePollIntervals(m.scheduler.MaxInterval())
}
return resourcePollIntervals{
pve: m.BasePollInterval(InstanceTypePVE),
diff --git a/internal/monitoring/scheduler.go b/internal/monitoring/scheduler.go
index a715eb1be0..d034cf380d 100644
--- a/internal/monitoring/scheduler.go
+++ b/internal/monitoring/scheduler.go
@@ -110,8 +110,11 @@ type AdaptiveScheduler struct {
lastPlan map[string]ScheduledTask
}
-// NewAdaptiveScheduler constructs a scheduler with safe defaults.
-func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interval IntervalSelector, enqueuer TaskEnqueuer) *AdaptiveScheduler {
+// normalizedSchedulerConfig fills unset or inconsistent bounds with the
+// defaults, as the scheduler does for the config it is given. Anything that
+// derives from the intervals a scheduler will select reads this, so it agrees
+// with the running scheduler on a partial or invalid configuration.
+func normalizedSchedulerConfig(cfg SchedulerConfig) SchedulerConfig {
if cfg.BaseInterval <= 0 {
cfg.BaseInterval = DefaultSchedulerConfig().BaseInterval
}
@@ -121,6 +124,12 @@ func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interv
if cfg.MaxInterval <= 0 || cfg.MaxInterval < cfg.MinInterval {
cfg.MaxInterval = DefaultSchedulerConfig().MaxInterval
}
+ return cfg
+}
+
+// NewAdaptiveScheduler constructs a scheduler with safe defaults.
+func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interval IntervalSelector, enqueuer TaskEnqueuer) *AdaptiveScheduler {
+ cfg = normalizedSchedulerConfig(cfg)
if staleness == nil {
staleness = noopStalenessSource{}
}
@@ -288,6 +297,17 @@ func (s *AdaptiveScheduler) DispatchDue(ctx context.Context, now time.Time, task
return due
}
+// MaxInterval is the longest interval the scheduler selects for an instance
+// without a fixed interval. A healthy instance's staleness score is near zero
+// right after each successful poll, so its cadence stretches toward this
+// bound; freshness judged by anything shorter flags healthy instances stale.
+func (s *AdaptiveScheduler) MaxInterval() time.Duration {
+ if s == nil {
+ return 0
+ }
+ return s.cfg.MaxInterval
+}
+
// LastScheduled returns the last recorded task for the given instance, if any.
func (s *AdaptiveScheduler) LastScheduled(instanceType InstanceType, instanceName string) (ScheduledTask, bool) {
if s == nil {