Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve history in metric storage guidance

pulse-triage[bot]5f595edf7e8a
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index b632e8d192..06ccd3191a 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -68,10 +68,14 @@ Do not reset signing or encryption keys as a troubleshooting step.
Path overrides:
- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.
-- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this
- for tmpfs-backed metrics history without moving secrets or config off the
- persistent data directory. It applies to the default org only; each
- non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
+- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path, not a
+ migration or write-reduction mode. Keep the selected directory on persistent
+ storage; a custom container path needs a persistent volume or bind mount too.
+ Changing the path does not move existing history. It applies to the default
+ org only; each non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
+ For an existing override or missing history, use the
+ [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning)
+ without restarting or changing the path just to diagnose it.
Multi-tenant layout:
- Default org uses the root data directory for backward compatibility.
diff --git a/docs/DOCKER.md b/docs/DOCKER.md
index 35f84e707b..bec0c2a873 100644
--- a/docs/DOCKER.md
+++ b/docs/DOCKER.md
@@ -125,7 +125,7 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid
| `ALLOWED_ORIGINS` | CORS allowed origin (`*` or a single origin). Empty = same-origin only. | *(unset)* |
| `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`) | `info` |
| `PULSE_DISABLE_DOCKER_UPDATE_ACTIONS` | Disable container image-update actions; does not disable start/stop/restart | `false` |
-| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`, useful with tmpfs | `/data/metrics.db` |
+| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`; keep it on persistent storage | `/data/metrics.db` |
| `PULSE_METRICS_ROLLUP_INTERVAL` | Metrics aggregation cadence; minimum 5 minutes | `15m` |
> **Tip**: Set `LOG_LEVEL=warn` to reduce log volume while still capturing important events.
@@ -135,19 +135,18 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid
> Docker administrators can read those values. Prefer bootstrap setup; never
> share full `docker inspect` or resolved Compose output.
-For SSD-sensitive installs, keep `/data` persistent and put only metrics
-history on tmpfs:
-
-```yaml
-services:
- pulse:
- environment:
- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
- tmpfs:
- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
-```
-
-Metrics history stored this way is lost on container restart.
+**Keep metric history on persistent storage as well as `/data`.** Moving
+`metrics.db` to tmpfs trades away retained observations; it does not repair
+excessive writes. A custom container path also needs its own persistent volume
+or bind mount: setting `PULSE_METRICS_DB_PATH` alone does not create one, and
+recreating the container discards an unmounted writable-layer store.
+
+If writes or disk use are unexpected, start with the
+[bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),
+not a storage change. For an existing override or history lost after recreation,
+see [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning).
+Do not restart, recreate the container, clear history or change retention just
+to diagnose it; preserve the original store and current deployment.
### Healthcheck and resource sizing
diff --git a/docs/METRICS_HISTORY.md b/docs/METRICS_HISTORY.md
index 39d94d5fe7..2bc86c5a47 100644
--- a/docs/METRICS_HISTORY.md
+++ b/docs/METRICS_HISTORY.md
@@ -157,62 +157,44 @@ After changing these values, restart Pulse.
## Advanced: Disk Write Tuning
-This is a deliberate storage trade-off, not a repair for missing history or
-unexplained writes. Changing the path opens a different history store; it does
-not migrate the existing history. Keep the original durable data and establish
-the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
-before changing storage.
-
-Pulse keeps metrics history on disk by default. SSD-sensitive installs can move
-only the metrics SQLite database without moving secrets or general config:
-
-```bash
-PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db
-```
-
-Use a dedicated directory owned by the account running Pulse. Do not place the
-database directly in a shared parent such as `/tmp/metrics.db`: Pulse secures
-the database directory to mode `0700` and will reject a directory owned by
-another user. On systemd installs, `PrivateTmp=true` also means a path below
-`/tmp` is private to the service and will not appear in the host's `/tmp`.
-
-The default systemd sandbox permits `/dev/shm/pulse` and Pulse can create that
-subdirectory itself. A dedicated mount elsewhere, such as `/mnt/ramdisk`, also
-needs an explicit writable-path grant:
-
-```bash
-sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse
-sudo systemctl edit pulse
-```
-
-Add this drop-in (replace `pulse` with the installed unit name if different):
-
-```ini
-[Service]
-ReadWritePaths=/mnt/ramdisk/pulse
-```
-
-Then set `PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db` in
-`/etc/pulse/.env` and restart Pulse. `ProtectSystem=strict` deliberately keeps
-paths outside the unit's writable allowlist read-only even when Unix ownership
-would otherwise permit writes.
-
-For Docker, mount a tmpfs at the selected directory and keep `/data` on a
-persistent volume:
-
-```yaml
-services:
- pulse:
- environment:
- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
- tmpfs:
- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
-```
-
-Using tmpfs loses that history when the RAM-backed mount is discarded, for
-example on a host reboot or container recreation. A service restart alone is
-not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains
-config, encrypted credentials, tokens, and other state that must remain durable.
+**Keep history on persistent storage.** SSD write pressure is a reason to
+identify the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),
+not to discard observations. Do not move history to tmpfs, clear it or shorten
+retention as a diagnostic workaround.
+
+`PULSE_METRICS_DB_PATH` remains a supported storage-location override for only
+the default organization's `metrics.db`; it does not move secrets or general
+configuration. **Changing the path opens a different history store; it does not
+migrate or merge the existing history.** An empty chart after a path change is
+not proof that the original data was deleted or that collection recovered.
+Keep the original store intact.
+
+For an existing override, distinguish these cases from the deployment's saved
+configuration and storage settings, using observations already available:
+
+| Selected storage | What survives, and what can be lost |
+|---|---|
+| Persistent directory, volume or bind mount | History can survive service restarts and container recreation while that same storage remains attached. Retention still applies; persistence is not a backup. |
+| Container writable layer without a volume or bind mount | A process restart can retain it, but container recreation discards it. Setting the path alone does not create a persistent mount. |
+| RAM-backed storage such as tmpfs or `/dev/shm` | History is lost when that mount is discarded, including host reboot or container recreation. A service restart alone is not guaranteed to clear it. |
+
+Do not use volatile storage for the whole data directory: it also holds
+configuration, encrypted credentials, tokens and other state that must remain
+durable. A separate durable metrics path must be writable by the actual Pulse
+service account and permitted by the deployment's existing sandbox. Use a
+dedicated private directory, not a shared parent such as `/tmp`; do not relax
+ownership or service confinement just to make a path work.
+
+**If history has already disappeared**, retain the last known chart range and
+the original update, recreation or reboot time if known. Compare the current
+configured path and mount with the previous deployment's saved settings; an
+unset override uses the [default storage location](#storage-location).
+Do not restart, recreate a container, switch paths again, copy an open database
+or delete the original store to test an explanation. Do not post configuration,
+databases or full container inspections. An unavailable previous path remains
+unknown; a new empty store cannot recover observations lost with a discarded
+mount. Use [Getting Help](TROUBLESHOOTING.md#-getting-help) with a relevant,
+locally redacted description of the storage type and sequence instead.
The aggregation cadence can also be lengthened when an install prefers fewer,
larger rollup writes over more frequent smaller writes:
diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.md
index b632e8d192..06ccd3191a 100644
--- a/frontend-modern/public/docs/CONFIGURATION.md
+++ b/frontend-modern/public/docs/CONFIGURATION.md
@@ -68,10 +68,14 @@ Do not reset signing or encryption keys as a troubleshooting step.
Path overrides:
- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.
-- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this
- for tmpfs-backed metrics history without moving secrets or config off the
- persistent data directory. It applies to the default org only; each
- non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
+- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path, not a
+ migration or write-reduction mode. Keep the selected directory on persistent
+ storage; a custom container path needs a persistent volume or bind mount too.
+ Changing the path does not move existing history. It applies to the default
+ org only; each non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
+ For an existing override or missing history, use the
+ [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning)
+ without restarting or changing the path just to diagnose it.
Multi-tenant layout:
- Default org uses the root data directory for backward compatibility.
diff --git a/frontend-modern/public/docs/DOCKER.md b/frontend-modern/public/docs/DOCKER.md
index 35f84e707b..bec0c2a873 100644
--- a/frontend-modern/public/docs/DOCKER.md
+++ b/frontend-modern/public/docs/DOCKER.md
@@ -125,7 +125,7 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid
| `ALLOWED_ORIGINS` | CORS allowed origin (`*` or a single origin). Empty = same-origin only. | *(unset)* |
| `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`) | `info` |
| `PULSE_DISABLE_DOCKER_UPDATE_ACTIONS` | Disable container image-update actions; does not disable start/stop/restart | `false` |
-| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`, useful with tmpfs | `/data/metrics.db` |
+| `PULSE_METRICS_DB_PATH` | Optional path for only `metrics.db`; keep it on persistent storage | `/data/metrics.db` |
| `PULSE_METRICS_ROLLUP_INTERVAL` | Metrics aggregation cadence; minimum 5 minutes | `15m` |
> **Tip**: Set `LOG_LEVEL=warn` to reduce log volume while still capturing important events.
@@ -135,19 +135,18 @@ Pulse is configured via the UI (`system.json`) with optional environment overrid
> Docker administrators can read those values. Prefer bootstrap setup; never
> share full `docker inspect` or resolved Compose output.
-For SSD-sensitive installs, keep `/data` persistent and put only metrics
-history on tmpfs:
-
-```yaml
-services:
- pulse:
- environment:
- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
- tmpfs:
- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
-```
-
-Metrics history stored this way is lost on container restart.
+**Keep metric history on persistent storage as well as `/data`.** Moving
+`metrics.db` to tmpfs trades away retained observations; it does not repair
+excessive writes. A custom container path also needs its own persistent volume
+or bind mount: setting `PULSE_METRICS_DB_PATH` alone does not create one, and
+recreating the container discards an unmounted writable-layer store.
+
+If writes or disk use are unexpected, start with the
+[bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),
+not a storage change. For an existing override or history lost after recreation,
+see [metrics storage checks](METRICS_HISTORY.md#advanced-disk-write-tuning).
+Do not restart, recreate the container, clear history or change retention just
+to diagnose it; preserve the original store and current deployment.
### Healthcheck and resource sizing
diff --git a/frontend-modern/public/docs/METRICS_HISTORY.md b/frontend-modern/public/docs/METRICS_HISTORY.md
index 39d94d5fe7..2bc86c5a47 100644
--- a/frontend-modern/public/docs/METRICS_HISTORY.md
+++ b/frontend-modern/public/docs/METRICS_HISTORY.md
@@ -157,62 +157,44 @@ After changing these values, restart Pulse.
## Advanced: Disk Write Tuning
-This is a deliberate storage trade-off, not a repair for missing history or
-unexplained writes. Changing the path opens a different history store; it does
-not migrate the existing history. Keep the original durable data and establish
-the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
-before changing storage.
-
-Pulse keeps metrics history on disk by default. SSD-sensitive installs can move
-only the metrics SQLite database without moving secrets or general config:
-
-```bash
-PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db
-```
-
-Use a dedicated directory owned by the account running Pulse. Do not place the
-database directly in a shared parent such as `/tmp/metrics.db`: Pulse secures
-the database directory to mode `0700` and will reject a directory owned by
-another user. On systemd installs, `PrivateTmp=true` also means a path below
-`/tmp` is private to the service and will not appear in the host's `/tmp`.
-
-The default systemd sandbox permits `/dev/shm/pulse` and Pulse can create that
-subdirectory itself. A dedicated mount elsewhere, such as `/mnt/ramdisk`, also
-needs an explicit writable-path grant:
-
-```bash
-sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse
-sudo systemctl edit pulse
-```
-
-Add this drop-in (replace `pulse` with the installed unit name if different):
-
-```ini
-[Service]
-ReadWritePaths=/mnt/ramdisk/pulse
-```
-
-Then set `PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db` in
-`/etc/pulse/.env` and restart Pulse. `ProtectSystem=strict` deliberately keeps
-paths outside the unit's writable allowlist read-only even when Unix ownership
-would otherwise permit writes.
-
-For Docker, mount a tmpfs at the selected directory and keep `/data` on a
-persistent volume:
-
-```yaml
-services:
- pulse:
- environment:
- PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
- tmpfs:
- - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
-```
-
-Using tmpfs loses that history when the RAM-backed mount is discarded, for
-example on a host reboot or container recreation. A service restart alone is
-not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains
-config, encrypted credentials, tokens, and other state that must remain durable.
+**Keep history on persistent storage.** SSD write pressure is a reason to
+identify the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth),
+not to discard observations. Do not move history to tmpfs, clear it or shorten
+retention as a diagnostic workaround.
+
+`PULSE_METRICS_DB_PATH` remains a supported storage-location override for only
+the default organization's `metrics.db`; it does not move secrets or general
+configuration. **Changing the path opens a different history store; it does not
+migrate or merge the existing history.** An empty chart after a path change is
+not proof that the original data was deleted or that collection recovered.
+Keep the original store intact.
+
+For an existing override, distinguish these cases from the deployment's saved
+configuration and storage settings, using observations already available:
+
+| Selected storage | What survives, and what can be lost |
+|---|---|
+| Persistent directory, volume or bind mount | History can survive service restarts and container recreation while that same storage remains attached. Retention still applies; persistence is not a backup. |
+| Container writable layer without a volume or bind mount | A process restart can retain it, but container recreation discards it. Setting the path alone does not create a persistent mount. |
+| RAM-backed storage such as tmpfs or `/dev/shm` | History is lost when that mount is discarded, including host reboot or container recreation. A service restart alone is not guaranteed to clear it. |
+
+Do not use volatile storage for the whole data directory: it also holds
+configuration, encrypted credentials, tokens and other state that must remain
+durable. A separate durable metrics path must be writable by the actual Pulse
+service account and permitted by the deployment's existing sandbox. Use a
+dedicated private directory, not a shared parent such as `/tmp`; do not relax
+ownership or service confinement just to make a path work.
+
+**If history has already disappeared**, retain the last known chart range and
+the original update, recreation or reboot time if known. Compare the current
+configured path and mount with the previous deployment's saved settings; an
+unset override uses the [default storage location](#storage-location).
+Do not restart, recreate a container, switch paths again, copy an open database
+or delete the original store to test an explanation. Do not post configuration,
+databases or full container inspections. An unavailable previous path remains
+unknown; a new empty store cannot recover observations lost with a discarded
+mount. Use [Getting Help](TROUBLESHOOTING.md#-getting-help) with a relevant,
+locally redacted description of the storage type and sequence instead.
The aggregation cadence can also be lengthened when an install prefers fewer,
larger rollup writes over more frequent smaller writes:
diff --git a/scripts/tests/test_metrics_storage_docs.py b/scripts/tests/test_metrics_storage_docs.py
new file mode 100644
index 0000000000..7144c270ac
--- /dev/null
+++ b/scripts/tests/test_metrics_storage_docs.py
@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+"""Keep storage help from turning write-pressure diagnosis into history loss.
+
+These are documentation safety checks, not a database migration or proof of
+installed persistence. The normal documentation/smoke harness discovers them.
+"""
+
+from pathlib import Path
+import re
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+GUIDES = ("DOCKER.md", "CONFIGURATION.md", "METRICS_HISTORY.md")
+
+
+def volatile_metrics_recipes(source):
+ blocks = re.findall(r"```[^\n]*\n(.*?)```", source, re.S)
+ return [block for block in blocks if re.search(
+ r"PULSE_METRICS_DB_PATH\s*[:=]\s*[\"']?/(?:dev/shm|mnt/ramdisk|metrics-tmpfs)/",
+ block,
+ )]
+
+
+class MetricsStorageDocsTest(unittest.TestCase):
+ def test_all_three_shipped_guides_match_the_canonical_documents(self):
+ for name in GUIDES:
+ with self.subTest(guide=name):
+ self.assertEqual((ROOT / "docs" / name).read_bytes(),
+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())
+
+ def test_copyable_examples_do_not_move_history_to_volatile_storage(self):
+ for name in GUIDES:
+ with self.subTest(guide=name):
+ self.assertEqual(volatile_metrics_recipes((ROOT / "docs" / name).read_text()), [])
+ # Cover the shell assignment and Compose forms that previously shipped.
+ for example in (
+ "```bash\nPULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db\n```",
+ "```bash\nPULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db\n```",
+ "```yaml\nenvironment:\n PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db\n```",
+ ):
+ with self.subTest(example=example):
+ self.assertEqual(len(volatile_metrics_recipes(example)), 1)
+
+ def test_storage_explanation_distinguishes_the_three_lifetimes(self):
+ text = (ROOT / "docs/METRICS_HISTORY.md").read_text().split(
+ "## Advanced: Disk Write Tuning\n", 1)[1].split("\n## ", 1)[0]
+ rows = re.findall(r"^\| (.+?) \| (.+?) \|$", text, re.M)
+ self.assertEqual(len(rows), 4) # Header and the three storage types.
+ persistent, writable, ram = [description for _, description in rows[1:]]
+ self.assertIn("same storage remains attached", persistent)
+ self.assertIn("persistence is not a backup", persistent)
+ self.assertIn("process restart can retain it", writable)
+ self.assertIn("container recreation discards it", writable)
+ self.assertIn("when that mount is discarded", ram)
+ self.assertIn("service restart alone is not guaranteed", ram)
+ normal = " ".join(text.split())
+ for boundary in ("supported storage-location override", "default organization",
+ "does not migrate or merge", "Keep the original store intact",
+ "Do not restart, recreate a container, switch paths again",
+ "copy an open database", "Do not post configuration",
+ "cannot recover observations lost"):
+ with self.subTest(boundary=boundary):
+ self.assertIn(boundary, normal)
+
+ def test_entry_points_preserve_the_override_and_link_to_storage_checks(self):
+ for name in ("DOCKER.md", "CONFIGURATION.md"):
+ text = (ROOT / "docs" / name).read_text()
+ with self.subTest(guide=name):
+ self.assertIn("PULSE_METRICS_DB_PATH", text)
+ self.assertIn("METRICS_HISTORY.md#advanced-disk-write-tuning", text)
+ self.assertIn("persistent volume", text)
+ history = (ROOT / "docs/METRICS_HISTORY.md").read_text()
+ self.assertIn("## Advanced: Disk Write Tuning", history)
+ self.assertIn("TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth", history)
+ self.assertIn("TROUBLESHOOTING.md#-getting-help", history)
+
+
+if __name__ == "__main__":
+ unittest.main()