Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

17.0 KB
Raw Blame

Metrics History (Persistent)

Pulse persists metrics history to disk so trend views and sparklines survive restarts.

Read a History panel

In current drawers, open the affected resource's Overview and use History range beside its charts. Proxmox guests and nodes, Docker hosts and resource drawers (including PBS) show stored charts there instead of a separate History tab. If your installed version still has a History tab, use that tab; do not upgrade or restart just to follow this guide. A resource without a supported history target may have no chart.

On a phone, drawers can show only Utilization and panels with active alerts. Use the Show … button below the charts to reveal the others, for example Show Network I/O, Disk I/O, Thermals. Its labels name the currently collapsed panels, so they vary by resource and alert. Show fewer charts collapses them again. Wider layouts show all panels and may have no button; older installed layouts may differ.

This button changes only the layout, preserving History range; it does not refresh history or collect new measurements. A collapsed panel is not missing data or measured zero. After revealing it, No stored history in this range means that panel has no plotted stored samples for the selected range, not zero usage or proof that collection stopped. Loading, an access error, a locked range and History unavailable are different states: retain the existing message and range. Do not refresh, run Diagnostics or restart just to reveal a chart.

These charts are separate from Alerts → History, table sparklines and current summary cards. A current card or a short trend preview does not establish the observations in the selected stored range. For a PBS host, use the Backup Server chart, not its datastore or the VM that happens to host it.

For a Proxmox VM, linking a Pulse Agent does not select agent I/O for the VM's live or stored Disk I/O chart. Passed-through storage can be busy while the Proxmox VM chart is quiet; compare the VM and agent measurement sources before treating that trace as an idle guest.

Start with the existing chart, not a restart or a storage change. A current reading does not prove that it was stored, and a selected time range does not prove continuous coverage. Window start and Window end bound the requested interval; they are not the first and last successful observations.

VM and LXC drawer panels can show these labels. Other charts and older Pulse versions may differ; if a time or source is not shown, keep it unknown rather than borrowing one from the current overview.

Display How to read it What it does not establish
Latest stored The newest stored observation time in this panel. Each value belongs to that time; a dash means no stored observation for that metric at that time. Not necessarily now, every metric's newest value, or another panel's time.
Stored while inspecting Values at the selected stored observation time. Not a live reading or an interpolated observation between points.
current, last known or freshness unknown with no stored history A separately qualified fallback reading. Keep its label and explanation. Not proof that the value was persisted or belongs to a past incident.
Single observation. No trend yet. One stored time in that panel. Not a flat trend or sustained recovery.
Missing observations Some series lack readings at other stored times in that panel; lines stop across those gaps. Not the cause or duration of a monitoring outage. Missing is not measured zero.
Shaded low–high band The recorded minimum and maximum within an aggregated bucket, when supplied; the line shows its average. Not continuous coverage, every original sample, or the duration of a spike.
peak while inspecting That metric's maximum in the inspected bucket, when the panel shows it beside the average. Not necessarily the highest value in the whole selected range or the value at an alert's exact time.

For example, CPU 22% at 12:02 and memory 95% at 12:01 are not one observation. At Latest stored · 12:02, memory is a dash if there is no memory point at 12:02; the earlier 95% remains inspectable. Network and disk I/O panels have their own times, not a shared snapshot of the guest.

An aggregated CPU bucket can show 22% average, 10% to 95%. The line at 22% does not rule out the recorded 95% peak, and the band does not say how long that peak lasted. No visible band may mean a single reading, equal bounds or unavailable bounds; it does not prove that usage stayed constant. Older panels may not show these bounds. Keep that limitation rather than inferring a peak from the line alone.

  1. Keep the same resource, metric and range when comparing existing readings. Inspect history, when offered, steps through stored times: focus the slider and use the arrow keys, Home or End, or move it by touch. Hover is another way to inspect; it is not required. Leaving the slider returns to the latest stored display.
  2. Keep the displayed observation time and qualification with the value. Stored points can be aggregated or downsampled; they are not a measurement of every instant in the selected range. A page refresh time is not the observation time. If an absolute time is unavailable, retain that limit.
  3. Compare an alert or Patrol finding with the original episode, not just today's overview. Today's source cannot identify the source of a past warning. Follow guest memory interpretation when unlike readings suggest memory pressure.

Use already available observations; do not run Diagnostics or guest-agent probes, restart a guest, repeat a backup, clear History or change retention to make a comparison agree. An unresponsive guest or an uncertain freeze/thaw operation needs Backup safety, not chart inspection. When reporting a display disagreement, keep the panel, time, range, labels and relevant values with consistent private aliases; do not share raw responses, credentials or a full network export.

Retention and accessible range

Storage retention and the range your running instance can serve are separate. Community supports up to 7 days; Pro and legacy Pro+ entitlements support up to 90 days. The panel uses the active runtime's history limit, not the length of time the database has existed. A longer retention setting does not grant access, and a licence cannot recover observations that were never stored.

Relay is no longer sold. Continuing Relay subscribers receive Pro features, including 90-day history, at their existing price. Older Relay payloads can still show a 14-day compatibility limit; that is not a plan to buy or proof that a continuing subscription has received its current entitlement. See plans and legacy subscriptions.

An API request beyond the active limit returns 402 Payment Required, not an empty history or a failed collector. Use an allowed range and the existing licence/range display first; do not buy another licence, reset history or reinstall merely to diagnose it. Storage defaults below do not guarantee that every resource has data for every allowed range.

Storage Location

Metrics history is stored in a SQLite database named metrics.db under the Pulse data directory:

  • systemd/LXC installs: typically /etc/pulse/metrics.db
  • Docker/Kubernetes installs: typically /data/metrics.db

Retention Model (Tiered)

Pulse keeps multiple resolutions of the same data, which allows longer history without storing raw samples forever:

  • Raw (high-resolution, short window)
  • Minute aggregates
  • Hourly aggregates
  • Daily aggregates

Default retention values (subject to change) are:

  • Raw: 2 hours
  • Minute: 24 hours
  • Hourly: 7 days
  • Daily: 90 days
  • Rollups: every 15 minutes by default, bounded so rollups still run well before raw samples expire

Advanced: Retention Tuning

Do not change retention or polling merely because a chart is empty or writes are high. First follow Troubleshooting and preserve the original observations. Shorter retention removes history; slower polling also changes how quickly monitoring can notice a problem.

Tiered retention is stored in system.json in the Pulse data directory:

  • systemd/LXC installs: typically /etc/pulse/system.json
  • Docker/Kubernetes installs: typically /data/system.json

Keys:

{
  "metricsRetentionRawHours": 2,
  "metricsRetentionMinuteHours": 24,
  "metricsRetentionHourlyDays": 7,
  "metricsRetentionDailyDays": 90
}

After changing these values, restart Pulse.

Advanced: Disk Write Tuning

Keep history on persistent storage. SSD write pressure is a reason to identify the writer using bounded performance checks, not to discard observations. Do not move history to tmpfs, clear it or shorten retention as a diagnostic workaround.

PULSE_METRICS_DB_PATH remains a supported storage-location override for only the default organization's metrics.db; it does not move secrets or general configuration. This is not a repair for missing history or unexplained writes. Changing the path opens a different history store; it does not migrate the existing history or merge stores. An empty chart after a path change is not proof that the original data was deleted or that collection recovered. Keep the original store intact.

For an existing override, distinguish these cases from the deployment's saved configuration and storage settings, using observations already available:

Selected storage What survives, and what can be lost
Persistent directory, volume or bind mount History can survive service restarts and container recreation while that same storage remains attached. Retention still applies; persistence is not a backup.
Container writable layer without a volume or bind mount A process restart can retain it, but container recreation discards it. Setting the path alone does not create a persistent mount.
RAM-backed storage such as tmpfs or /dev/shm History is lost when that mount is discarded, including host reboot or container recreation. A service restart alone is not guaranteed to clear it.

Do not use volatile storage for the whole data directory: it also holds configuration, encrypted credentials, tokens and other state that must remain durable. A separate durable metrics path must be writable by the actual Pulse service account and permitted by the deployment's existing sandbox. Use a dedicated private directory, not a shared parent such as /tmp; do not relax ownership or service confinement just to make a path work.

If history has already disappeared, retain the last known chart range and the original update, recreation or reboot time if known. Compare the current configured path and mount with the previous deployment's saved settings; an unset override uses the default storage location. Do not restart, recreate a container, switch paths again, copy an open database or delete the original store to test an explanation. Do not post configuration, databases or full container inspections. An unavailable previous path remains unknown; a new empty store cannot recover observations lost with a discarded mount. Use Getting Help with a relevant, locally redacted description of the storage type and sequence instead.

The aggregation cadence can also be lengthened when an install prefers fewer, larger rollup writes over more frequent smaller writes:

PULSE_METRICS_ROLLUP_INTERVAL=30m

Values below 5 minutes are ignored. Values longer than half of the raw-retention window are capped by the metrics store so raw samples are still rolled up before retention pruning can remove them.

API Access

Pulse exposes the persistent metrics store via:

  • GET /api/metrics-store/stats
  • GET /api/metrics-store/history

These endpoints require authentication with the monitoring:read scope.

History Query Parameters

GET /api/metrics-store/history supports:

  • resourceType (required): node, storage, agent, disk, k8s (or the Kubernetes kinds pod, k8s-cluster, k8s-node, k8s-deployment, which read the same history), vm, system-container, oci-container, app-container, or docker-host
  • resourceId (required): the source identifier used by that chart, not its display name (for Proxmox guests this is normally instance:node:vmid)
  • metric (optional): cpu, memory, disk, etc. Omit to return all metrics for the resource.
  • range (optional): 1h, 6h, 12h, 24h, 1d, 7d, 30d, 90d (default 24h; duration strings also accepted)
  • maxPoints (optional): Downsample to a target number of points

For a one-off check, open the affected resource's History panel in your signed-in Pulse browser. If an API read is needed, use the path and query from that panel's request on the same instance; do not extract its session cookie, use Copy as cURL, or share a full network export. /api/metrics-store/stats describes the store overall, not whether this resource has the requested metric.

For curl, follow API authentication to prepare the private header file with only monitoring:read and define its pulse_api helper in the same Bash session. It is example code, not an installed Pulse command. The default helper runs on the Pulse host; for remote access, follow that guide's HTTPS-origin and verified-CA instructions. Then make one read-only request (curl 7.76 or later):

pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'

Replace the example type and ID with the chart's values, URL-encoding each query value (: becomes %3A; an ID's & must be %26, not another query parameter). Do not substitute a hostname or strip a provider prefix to guess an ID. The helper saves each response, including an error body, in a new owner-only file and prints only the HTTP status and file location. Open that file privately; do not print or post it. HTTP 401, 402, 403 and server errors, redirects, or incomplete transport are not an empty history or zero usage. There is no automatic retry. Never put a token in a command, URL or report; share only the relevant manually redacted error and bounded observations.

With metric=cpu (or another explicit metric), the response has a points array. Omitting metric instead returns a metrics object keyed by metric name, not the same single-series shape. Inspect timestamps and source too: store is persisted history; memory or live fallback does not establish durable historical coverage. An HTTP 200 with no points proves neither zero usage nor successful collection, and a non-zero current value does not prove every historical interval is present.

History limit: Requests beyond the active runtime's allowed range return 402 Payment Required. See retention and accessible range; stored retention is not access or proof of collected observations. Older examples: container, dockerHost, dockerContainer, guest and docker are not current resourceType query values. Use the chart's current type rather than a display label or internal storage name.

Troubleshooting

  • Missing or stale history: retain the affected panel, metric, range, collection time and current server/agent versions. Check the connection's last normal poll or agent report in Settings → Infrastructure, without running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw. Compare the panel's actual request, HTTP result, point timestamps and source. A valid connection test, another resource's graph or the existence of metrics.db does not establish this metric's collection or display.
  • Access or query error: keep it separate from an empty result. Check the selected organisation, monitoring:read access, exact query type/ID and the requested range. A 402 is a history-range entitlement failure, not evidence that storage is empty; try an allowed range without buying another licence or resetting data.
  • Storage warning: inspect a bounded existing startup/runtime error and the deployment's persistent mount and service-account access privately. Follow the safe troubleshooting guide for high writes or space pressure. Do not delete history, edit a database, shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.
  • Report still needed: share the affected metric/range, versions, whether normal collection advances, and a manually redacted error; keep resource identities, response files and databases private. Follow Getting Help, not a full response dump.