Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve history in metric storage guidance
pulse-triage[bot]5f595edf7e8a
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex b632e8d192..06ccd3191a 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -68,10 +68,14 @@ Do not reset signing or encryption keys as a troubleshooting step.Path overrides:- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.-- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this- for tmpfs-backed metrics history without moving secrets or config off the- persistent data directory. It applies to the default org only; each- non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.+- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path, not a+ migration or write-reduction mode. Keep the selected directory on persistent+ storage; a custom container path needs a persistent volume or bind mount too.+ Changing the path does not move existing history. It applies to the default+ org only; each non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.+ For an existing override or missing history, use the+ [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning)+ without restarting or changing the path just to diagnose it.Multi-tenant layout:- Default org uses the root data directory for backward compatibility.diff --git a/docs/DOCKER.md b/docs/DOCKER.mdindex 35f84e707b..bec0c2a873 100644--- a/docs/DOCKER.md+++ b/docs/DOCKER.md@@ -125,7 +125,7 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid| `ALLOWED_ORIGINS` | CORS allowed origin (`*` or a single origin). Empty = same-origin only. | *(unset)* || `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`) | `info` || `PULSE_DISABLE_DOCKER_UPDATE_ACTIONS` | Disable container image-update actions; does not disable start/stop/restart | `false` |-| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`, useful with tmpfs | `/data/metrics.db` |+| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`; keep it on persistent storage | `/data/metrics.db` || `PULSE_METRICS_ROLLUP_INTERVAL` | Metrics aggregation cadence; minimum 5 minutes | `15m` |> **Tip**: Set `LOG_LEVEL=warn` to reduce log volume while still capturing important events.@@ -135,19 +135,18 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid> Docker administrators can read those values. Prefer bootstrap setup; never> share full `docker inspect` or resolved Compose output.-For SSD-sensitive installs, keep `/data` persistent and put only metrics-history on tmpfs:--```yaml-services:- pulse:- environment:- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db- tmpfs:- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700-```--Metrics history stored this way is lost on container restart.+**Keep metric history on persistent storage as well as `/data`.** Moving+`metrics.db` to tmpfs trades away retained observations; it does not repair+excessive writes. A custom container path also needs its own persistent volume+or bind mount: setting `PULSE_METRICS_DB_PATH` alone does not create one, and+recreating the container discards an unmounted writable-layer store.++If writes or disk use are unexpected, start with the+[bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),+not a storage change. For an existing override or history lost after recreation,+see [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning).+Do not restart, recreate the container, clear history or change retention just+to diagnose it; preserve the original store and current deployment.### Healthcheck and resource sizingdiff --git a/docs/METRICS_HISTORY.md b/docs/METRICS_HISTORY.mdindex 39d94d5fe7..2bc86c5a47 100644--- a/docs/METRICS_HISTORY.md+++ b/docs/METRICS_HISTORY.md@@ -157,62 +157,44 @@ After changing these values, restart Pulse.## Advanced: Disk Write Tuning-This is a deliberate storage trade-off, not a repair for missing history or-unexplained writes. Changing the path opens a different history store; it does-not migrate the existing history. Keep the original durable data and establish-the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)-before changing storage.--Pulse keeps metrics history on disk by default. SSD-sensitive installs can move-only the metrics SQLite database without moving secrets or general config:--```bash-PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db-```--Use a dedicated directory owned by the account running Pulse. Do not place the-database directly in a shared parent such as `/tmp/metrics.db`: Pulse secures-the database directory to mode `0700` and will reject a directory owned by-another user. On systemd installs, `PrivateTmp=true` also means a path below-`/tmp` is private to the service and will not appear in the host's `/tmp`.--The default systemd sandbox permits `/dev/shm/pulse` and Pulse can create that-subdirectory itself. A dedicated mount elsewhere, such as `/mnt/ramdisk`, also-needs an explicit writable-path grant:--```bash-sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse-sudo systemctl edit pulse-```--Add this drop-in (replace `pulse` with the installed unit name if different):--```ini-[Service]-ReadWritePaths=/mnt/ramdisk/pulse-```--Then set `PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db` in-`/etc/pulse/.env` and restart Pulse. `ProtectSystem=strict` deliberately keeps-paths outside the unit's writable allowlist read-only even when Unix ownership-would otherwise permit writes.--For Docker, mount a tmpfs at the selected directory and keep `/data` on a-persistent volume:--```yaml-services:- pulse:- environment:- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db- tmpfs:- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700-```--Using tmpfs loses that history when the RAM-backed mount is discarded, for-example on a host reboot or container recreation. A service restart alone is-not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains-config, encrypted credentials, tokens, and other state that must remain durable.+**Keep history on persistent storage.** SSD write pressure is a reason to+identify the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),+not to discard observations. Do not move history to tmpfs, clear it or shorten+retention as a diagnostic workaround.++`PULSE_METRICS_DB_PATH` remains a supported storage-location override for only+the default organization's `metrics.db`; it does not move secrets or general+configuration. **Changing the path opens a different history store; it does not+migrate or merge the existing history.** An empty chart after a path change is+not proof that the original data was deleted or that collection recovered.+Keep the original store intact.++For an existing override, distinguish these cases from the deployment's saved+configuration and storage settings, using observations already available:++| Selected storage | What survives, and what can be lost |+|---|---|+| Persistent directory, volume or bind mount | History can survive service restarts and container recreation while that same storage remains attached. Retention still applies; persistence is not a backup. |+| Container writable layer without a volume or bind mount | A process restart can retain it, but container recreation discards it. Setting the path alone does not create a persistent mount. |+| RAM-backed storage such as tmpfs or `/dev/shm` | History is lost when that mount is discarded, including host reboot or container recreation. A service restart alone is not guaranteed to clear it. |++Do not use volatile storage for the whole data directory: it also holds+configuration, encrypted credentials, tokens and other state that must remain+durable. A separate durable metrics path must be writable by the actual Pulse+service account and permitted by the deployment's existing sandbox. Use a+dedicated private directory, not a shared parent such as `/tmp`; do not relax+ownership or service confinement just to make a path work.++**If history has already disappeared**, retain the last known chart range and+the original update, recreation or reboot time if known. Compare the current+configured path and mount with the previous deployment's saved settings; an+unset override uses the [default storage location](#storage-location).+Do not restart, recreate a container, switch paths again, copy an open database+or delete the original store to test an explanation. Do not post configuration,+databases or full container inspections. An unavailable previous path remains+unknown; a new empty store cannot recover observations lost with a discarded+mount. Use [Getting Help](TROUBLESHOOTING.md#-getting-help) with a relevant,+locally redacted description of the storage type and sequence instead.The aggregation cadence can also be lengthened when an install prefers fewer,larger rollup writes over more frequent smaller writes:diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.mdindex b632e8d192..06ccd3191a 100644--- a/frontend-modern/public/docs/CONFIGURATION.md+++ b/frontend-modern/public/docs/CONFIGURATION.md@@ -68,10 +68,14 @@ Do not reset signing or encryption keys as a troubleshooting step.Path overrides:- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.-- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this- for tmpfs-backed metrics history without moving secrets or config off the- persistent data directory. It applies to the default org only; each- non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.+- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path, not a+ migration or write-reduction mode. Keep the selected directory on persistent+ storage; a custom container path needs a persistent volume or bind mount too.+ Changing the path does not move existing history. It applies to the default+ org only; each non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.+ For an existing override or missing history, use the+ [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning)+ without restarting or changing the path just to diagnose it.Multi-tenant layout:- Default org uses the root data directory for backward compatibility.diff --git a/frontend-modern/public/docs/DOCKER.md b/frontend-modern/public/docs/DOCKER.mdindex 35f84e707b..bec0c2a873 100644--- a/frontend-modern/public/docs/DOCKER.md+++ b/frontend-modern/public/docs/DOCKER.md@@ -125,7 +125,7 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid| `ALLOWED_ORIGINS` | CORS allowed origin (`*` or a single origin). Empty = same-origin only. | *(unset)* || `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`) | `info` || `PULSE_DISABLE_DOCKER_UPDATE_ACTIONS` | Disable container image-update actions; does not disable start/stop/restart | `false` |-| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`, useful with tmpfs | `/data/metrics.db` |+| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`; keep it on persistent storage | `/data/metrics.db` || `PULSE_METRICS_ROLLUP_INTERVAL` | Metrics aggregation cadence; minimum 5 minutes | `15m` |> **Tip**: Set `LOG_LEVEL=warn` to reduce log volume while still capturing important events.@@ -135,19 +135,18 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid> Docker administrators can read those values. Prefer bootstrap setup; never> share full `docker inspect` or resolved Compose output.-For SSD-sensitive installs, keep `/data` persistent and put only metrics-history on tmpfs:--```yaml-services:- pulse:- environment:- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db- tmpfs:- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700-```--Metrics history stored this way is lost on container restart.+**Keep metric history on persistent storage as well as `/data`.** Moving+`metrics.db` to tmpfs trades away retained observations; it does not repair+excessive writes. A custom container path also needs its own persistent volume+or bind mount: setting `PULSE_METRICS_DB_PATH` alone does not create one, and+recreating the container discards an unmounted writable-layer store.++If writes or disk use are unexpected, start with the+[bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),+not a storage change. For an existing override or history lost after recreation,+see [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning).+Do not restart, recreate the container, clear history or change retention just+to diagnose it; preserve the original store and current deployment.### Healthcheck and resource sizingdiff --git a/frontend-modern/public/docs/METRICS_HISTORY.md b/frontend-modern/public/docs/METRICS_HISTORY.mdindex 39d94d5fe7..2bc86c5a47 100644--- a/frontend-modern/public/docs/METRICS_HISTORY.md+++ b/frontend-modern/public/docs/METRICS_HISTORY.md@@ -157,62 +157,44 @@ After changing these values, restart Pulse.## Advanced: Disk Write Tuning-This is a deliberate storage trade-off, not a repair for missing history or-unexplained writes. Changing the path opens a different history store; it does-not migrate the existing history. Keep the original durable data and establish-the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)-before changing storage.--Pulse keeps metrics history on disk by default. SSD-sensitive installs can move-only the metrics SQLite database without moving secrets or general config:--```bash-PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db-```--Use a dedicated directory owned by the account running Pulse. Do not place the-database directly in a shared parent such as `/tmp/metrics.db`: Pulse secures-the database directory to mode `0700` and will reject a directory owned by-another user. On systemd installs, `PrivateTmp=true` also means a path below-`/tmp` is private to the service and will not appear in the host's `/tmp`.--The default systemd sandbox permits `/dev/shm/pulse` and Pulse can create that-subdirectory itself. A dedicated mount elsewhere, such as `/mnt/ramdisk`, also-needs an explicit writable-path grant:--```bash-sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse-sudo systemctl edit pulse-```--Add this drop-in (replace `pulse` with the installed unit name if different):--```ini-[Service]-ReadWritePaths=/mnt/ramdisk/pulse-```--Then set `PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db` in-`/etc/pulse/.env` and restart Pulse. `ProtectSystem=strict` deliberately keeps-paths outside the unit's writable allowlist read-only even when Unix ownership-would otherwise permit writes.--For Docker, mount a tmpfs at the selected directory and keep `/data` on a-persistent volume:--```yaml-services:- pulse:- environment:- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db- tmpfs:- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700-```--Using tmpfs loses that history when the RAM-backed mount is discarded, for-example on a host reboot or container recreation. A service restart alone is-not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains-config, encrypted credentials, tokens, and other state that must remain durable.+**Keep history on persistent storage.** SSD write pressure is a reason to+identify the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),+not to discard observations. Do not move history to tmpfs, clear it or shorten+retention as a diagnostic workaround.++`PULSE_METRICS_DB_PATH` remains a supported storage-location override for only+the default organization's `metrics.db`; it does not move secrets or general+configuration. **Changing the path opens a different history store; it does not+migrate or merge the existing history.** An empty chart after a path change is+not proof that the original data was deleted or that collection recovered.+Keep the original store intact.++For an existing override, distinguish these cases from the deployment's saved+configuration and storage settings, using observations already available:++| Selected storage | What survives, and what can be lost |+|---|---|+| Persistent directory, volume or bind mount | History can survive service restarts and container recreation while that same storage remains attached. Retention still applies; persistence is not a backup. |+| Container writable layer without a volume or bind mount | A process restart can retain it, but container recreation discards it. Setting the path alone does not create a persistent mount. |+| RAM-backed storage such as tmpfs or `/dev/shm` | History is lost when that mount is discarded, including host reboot or container recreation. A service restart alone is not guaranteed to clear it. |++Do not use volatile storage for the whole data directory: it also holds+configuration, encrypted credentials, tokens and other state that must remain+durable. A separate durable metrics path must be writable by the actual Pulse+service account and permitted by the deployment's existing sandbox. Use a+dedicated private directory, not a shared parent such as `/tmp`; do not relax+ownership or service confinement just to make a path work.++**If history has already disappeared**, retain the last known chart range and+the original update, recreation or reboot time if known. Compare the current+configured path and mount with the previous deployment's saved settings; an+unset override uses the [default storage location](#storage-location).+Do not restart, recreate a container, switch paths again, copy an open database+or delete the original store to test an explanation. Do not post configuration,+databases or full container inspections. An unavailable previous path remains+unknown; a new empty store cannot recover observations lost with a discarded+mount. Use [Getting Help](TROUBLESHOOTING.md#-getting-help) with a relevant,+locally redacted description of the storage type and sequence instead.The aggregation cadence can also be lengthened when an install prefers fewer,larger rollup writes over more frequent smaller writes:diff --git a/scripts/tests/test_metrics_storage_docs.py b/scripts/tests/test_metrics_storage_docs.pynew file mode 100644index 0000000000..7144c270ac--- /dev/null+++ b/scripts/tests/test_metrics_storage_docs.py@@ -0,0 +1,80 @@+#!/usr/bin/env python3+"""Keep storage help from turning write-pressure diagnosis into history loss.++These are documentation safety checks, not a database migration or proof of+installed persistence. The normal documentation/smoke harness discovers them.+"""++from pathlib import Path+import re+import unittest+++ROOT = Path(__file__).resolve().parents[2]+GUIDES = ("DOCKER.md", "CONFIGURATION.md", "METRICS_HISTORY.md")+++def volatile_metrics_recipes(source):+ blocks = re.findall(r"```[^\n]*\n(.*?)```", source, re.S)+ return [block for block in blocks if re.search(+ r"PULSE_METRICS_DB_PATH\s*[:=]\s*[\"']?/(?:dev/shm|mnt/ramdisk|metrics-tmpfs)/",+ block,+ )]+++class MetricsStorageDocsTest(unittest.TestCase):+ def test_all_three_shipped_guides_match_the_canonical_documents(self):+ for name in GUIDES:+ with self.subTest(guide=name):+ self.assertEqual((ROOT / "docs" / name).read_bytes(),+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())++ def test_copyable_examples_do_not_move_history_to_volatile_storage(self):+ for name in GUIDES:+ with self.subTest(guide=name):+ self.assertEqual(volatile_metrics_recipes((ROOT / "docs" / name).read_text()), [])+ # Cover the shell assignment and Compose forms that previously shipped.+ for example in (+ "```bash\nPULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db\n```",+ "```bash\nPULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db\n```",+ "```yaml\nenvironment:\n PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db\n```",+ ):+ with self.subTest(example=example):+ self.assertEqual(len(volatile_metrics_recipes(example)), 1)++ def test_storage_explanation_distinguishes_the_three_lifetimes(self):+ text = (ROOT / "docs/METRICS_HISTORY.md").read_text().split(+ "## Advanced: Disk Write Tuning\n", 1)[1].split("\n## ", 1)[0]+ rows = re.findall(r"^\| (.+?) \| (.+?) \|$", text, re.M)+ self.assertEqual(len(rows), 4) # Header and the three storage types.+ persistent, writable, ram = [description for _, description in rows[1:]]+ self.assertIn("same storage remains attached", persistent)+ self.assertIn("persistence is not a backup", persistent)+ self.assertIn("process restart can retain it", writable)+ self.assertIn("container recreation discards it", writable)+ self.assertIn("when that mount is discarded", ram)+ self.assertIn("service restart alone is not guaranteed", ram)+ normal = " ".join(text.split())+ for boundary in ("supported storage-location override", "default organization",+ "does not migrate or merge", "Keep the original store intact",+ "Do not restart, recreate a container, switch paths again",+ "copy an open database", "Do not post configuration",+ "cannot recover observations lost"):+ with self.subTest(boundary=boundary):+ self.assertIn(boundary, normal)++ def test_entry_points_preserve_the_override_and_link_to_storage_checks(self):+ for name in ("DOCKER.md", "CONFIGURATION.md"):+ text = (ROOT / "docs" / name).read_text()+ with self.subTest(guide=name):+ self.assertIn("PULSE_METRICS_DB_PATH", text)+ self.assertIn("METRICS_HISTORY.md#advanced-disk-write-tuning", text)+ self.assertIn("persistent volume", text)+ history = (ROOT / "docs/METRICS_HISTORY.md").read_text()+ self.assertIn("## Advanced: Disk Write Tuning", history)+ self.assertIn("TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth", history)+ self.assertIn("TROUBLESHOOTING.md#-getting-help", history)+++if __name__ == "__main__":+ unittest.main()