Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep history diagnostics private and match current query types
pulse-triage[bot]c42075806891
diff --git a/docs/API.md b/docs/API.mdindex 7c5aea6f7a..296d4cc116 100644--- a/docs/API.md+++ b/docs/API.md@@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed).Returns historical metric series for a resource and time range.Query params:-- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`+- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host`- `resourceId` (required)- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)- `maxPoints` (optional): Downsample to a target number of points> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.-> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.+An explicit `metric` returns a `points` array; omitting it returns a `metrics`+object keyed by metric name. Use the source ID and type from the affected+chart's request, not its display name or an internal store type. Older+`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values+are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for+private one-shot reads, response sources and empty-history troubleshooting.---diff --git a/docs/METRICS_HISTORY.md b/docs/METRICS_HISTORY.mdindex 43ee4e4c35..9165500d6d 100644--- a/docs/METRICS_HISTORY.md+++ b/docs/METRICS_HISTORY.md@@ -29,6 +29,11 @@ Default retention values (subject to change) are:## Advanced: Retention Tuning+Do not change retention or polling merely because a chart is empty or writes+are high. First follow [Troubleshooting](#troubleshooting) and preserve the+original observations. Shorter retention removes history; slower polling also+changes how quickly monitoring can notice a problem.+Tiered retention is stored in `system.json` in the Pulse data directory:- **systemd/LXC installs**: typically `/etc/pulse/system.json`@@ -49,6 +54,12 @@ After changing these values, restart Pulse.## Advanced: Disk Write Tuning+This is a deliberate storage trade-off, not a repair for missing history or+unexplained writes. Changing the path opens a different history store; it does+not migrate the existing history. Keep the original durable data and establish+the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+before changing storage.+Pulse keeps metrics history on disk by default. SSD-sensitive installs can moveonly the metrics SQLite database without moving secrets or general config:@@ -95,9 +106,10 @@ services:- /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700```-Using tmpfs makes metrics history ephemeral across restarts. It should not be-used for `/data`, because `/data` also contains config, encrypted credentials,-tokens, and other state that must remain durable.+Using tmpfs loses that history when the RAM-backed mount is discarded, for+example on a host reboot or container recreation. A service restart alone is+not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains+config, encrypted credentials, tokens, and other state that must remain durable.The aggregation cadence can also be lengthened when an install prefers fewer,larger rollup writes over more frequent smaller writes:@@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope.`GET /api/metrics-store/history` supports:-- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`-- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`)+- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`,+ `system-container`, `oci-container`, `app-container`, or `docker-host`+- `resourceId` (required): the source identifier used by that chart, not its+ display name (for Proxmox guests this is normally `instance:node:vmid`)- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource.- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)- `maxPoints` (optional): Downsample to a target number of points-For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse-browser; do not extract its session cookie. For curl, prepare the private-header file described in [API authentication](API.md#-authentication), using a-token with only `monitoring:read`. The example requires curl 7.76 or later and-runs **on the Pulse host**:+For a one-off check, open the affected resource's History panel in your signed-in+Pulse browser. If an API read is needed, use the path and query from that+panel's request on the same instance; do not extract its session cookie, use+**Copy as cURL**, or share a full network export. `/api/metrics-store/stats`+describes the store overall, not whether this resource has the requested metric.++For curl, follow [API authentication](API.md#-authentication) to prepare the+private header file with only `monitoring:read` and define its `pulse_api`+helper **in the same Bash session**. It is example code, not an installed Pulse+command. The default helper runs **on the Pulse host**; for remote access,+follow that guide's HTTPS-origin and verified-CA instructions. Then make one+read-only request (curl 7.76 or later):```bash-curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \- --header "@$HOME/.config/pulse/api-header" --get \- --data-urlencode 'resourceType=vm' \- --data-urlencode 'resourceId=pve1:node1:100' \- --data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \- http://127.0.0.1:7655/api/metrics-store/history+pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'```-Replace the example resource ID with the ID reported by your instance. For a-remote request, use your Pulse HTTPS address and keep certificate verification-enabled. Keep `--disable` first to ignore local curl defaults that could enable-credential-bearing traces. HTTP errors, including 401, 402 and 403, return a-non-zero exit; do not treat the error body as history. Never put a token in a-command, URL or report, and share only the relevant redacted error rather than-the whole infrastructure response.+Replace the example type and ID with the chart's values, URL-encoding each+query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query+parameter). Do not substitute a hostname or strip a provider prefix to guess+an ID. The helper saves each response, including an error body, in a new+owner-only file and prints only the HTTP status and file location. Open that+file privately; do not print or post it. HTTP 401, 402, 403 and server errors,+redirects, or incomplete transport are not an empty history or zero usage.+There is no automatic retry. Never put a token in a command, URL or report;+share only the relevant manually redacted error and bounded observations.++With `metric=cpu` (or another explicit metric), the response has a `points`+array. Omitting `metric` instead returns a `metrics` object keyed by metric+name, not the same single-series shape. Inspect timestamps and `source` too:+`store` is persisted history; `memory` or `live` fallback does not establish+durable historical coverage. An HTTP 200 with no points proves neither zero+usage nor successful collection, and a non-zero current value does not prove+every historical interval is present.> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.-> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.+> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and+> `docker` are not current `resourceType` query values. Use the chart's current+> type rather than a display label or internal storage name.## Troubleshooting-- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists.-- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse.+- **Missing or stale history**: retain the affected panel, metric, range,+ collection time and current server/agent versions. Check the connection's+ last normal poll or agent report in **Settings → Infrastructure**, without+ running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw.+ Compare the panel's actual request, HTTP result, point timestamps and source.+ A valid connection test, another resource's graph or the existence of+ `metrics.db` does not establish this metric's collection or display.+- **Access or query error**: keep it separate from an empty result. Check the+ selected organisation, `monitoring:read` access, exact query type/ID and the+ requested range. A 402 is a history-range entitlement failure, not evidence+ that storage is empty; try an allowed range without buying another licence+ or resetting data.+- **Storage warning**: inspect a bounded existing startup/runtime error and+ the deployment's persistent mount and service-account access privately.+ Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+ for high writes or space pressure. Do not delete history, edit a database,+ shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.+- **Report still needed**: share the affected metric/range, versions,+ whether normal collection advances, and a manually redacted error; keep+ resource identities, response files and databases private. Follow+ [Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump.diff --git a/frontend-modern/public/docs/API.md b/frontend-modern/public/docs/API.mdindex 7c5aea6f7a..296d4cc116 100644--- a/frontend-modern/public/docs/API.md+++ b/frontend-modern/public/docs/API.md@@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed).Returns historical metric series for a resource and time range.Query params:-- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`+- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host`- `resourceId` (required)- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)- `maxPoints` (optional): Downsample to a target number of points> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.-> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.+An explicit `metric` returns a `points` array; omitting it returns a `metrics`+object keyed by metric name. Use the source ID and type from the affected+chart's request, not its display name or an internal store type. Older+`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values+are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for+private one-shot reads, response sources and empty-history troubleshooting.---diff --git a/frontend-modern/public/docs/METRICS_HISTORY.md b/frontend-modern/public/docs/METRICS_HISTORY.mdindex 43ee4e4c35..9165500d6d 100644--- a/frontend-modern/public/docs/METRICS_HISTORY.md+++ b/frontend-modern/public/docs/METRICS_HISTORY.md@@ -29,6 +29,11 @@ Default retention values (subject to change) are:## Advanced: Retention Tuning+Do not change retention or polling merely because a chart is empty or writes+are high. First follow [Troubleshooting](#troubleshooting) and preserve the+original observations. Shorter retention removes history; slower polling also+changes how quickly monitoring can notice a problem.+Tiered retention is stored in `system.json` in the Pulse data directory:- **systemd/LXC installs**: typically `/etc/pulse/system.json`@@ -49,6 +54,12 @@ After changing these values, restart Pulse.## Advanced: Disk Write Tuning+This is a deliberate storage trade-off, not a repair for missing history or+unexplained writes. Changing the path opens a different history store; it does+not migrate the existing history. Keep the original durable data and establish+the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+before changing storage.+Pulse keeps metrics history on disk by default. SSD-sensitive installs can moveonly the metrics SQLite database without moving secrets or general config:@@ -95,9 +106,10 @@ services:- /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700```-Using tmpfs makes metrics history ephemeral across restarts. It should not be-used for `/data`, because `/data` also contains config, encrypted credentials,-tokens, and other state that must remain durable.+Using tmpfs loses that history when the RAM-backed mount is discarded, for+example on a host reboot or container recreation. A service restart alone is+not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains+config, encrypted credentials, tokens, and other state that must remain durable.The aggregation cadence can also be lengthened when an install prefers fewer,larger rollup writes over more frequent smaller writes:@@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope.`GET /api/metrics-store/history` supports:-- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`-- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`)+- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`,+ `system-container`, `oci-container`, `app-container`, or `docker-host`+- `resourceId` (required): the source identifier used by that chart, not its+ display name (for Proxmox guests this is normally `instance:node:vmid`)- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource.- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)- `maxPoints` (optional): Downsample to a target number of points-For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse-browser; do not extract its session cookie. For curl, prepare the private-header file described in [API authentication](API.md#-authentication), using a-token with only `monitoring:read`. The example requires curl 7.76 or later and-runs **on the Pulse host**:+For a one-off check, open the affected resource's History panel in your signed-in+Pulse browser. If an API read is needed, use the path and query from that+panel's request on the same instance; do not extract its session cookie, use+**Copy as cURL**, or share a full network export. `/api/metrics-store/stats`+describes the store overall, not whether this resource has the requested metric.++For curl, follow [API authentication](API.md#-authentication) to prepare the+private header file with only `monitoring:read` and define its `pulse_api`+helper **in the same Bash session**. It is example code, not an installed Pulse+command. The default helper runs **on the Pulse host**; for remote access,+follow that guide's HTTPS-origin and verified-CA instructions. Then make one+read-only request (curl 7.76 or later):```bash-curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \- --header "@$HOME/.config/pulse/api-header" --get \- --data-urlencode 'resourceType=vm' \- --data-urlencode 'resourceId=pve1:node1:100' \- --data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \- http://127.0.0.1:7655/api/metrics-store/history+pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'```-Replace the example resource ID with the ID reported by your instance. For a-remote request, use your Pulse HTTPS address and keep certificate verification-enabled. Keep `--disable` first to ignore local curl defaults that could enable-credential-bearing traces. HTTP errors, including 401, 402 and 403, return a-non-zero exit; do not treat the error body as history. Never put a token in a-command, URL or report, and share only the relevant redacted error rather than-the whole infrastructure response.+Replace the example type and ID with the chart's values, URL-encoding each+query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query+parameter). Do not substitute a hostname or strip a provider prefix to guess+an ID. The helper saves each response, including an error body, in a new+owner-only file and prints only the HTTP status and file location. Open that+file privately; do not print or post it. HTTP 401, 402, 403 and server errors,+redirects, or incomplete transport are not an empty history or zero usage.+There is no automatic retry. Never put a token in a command, URL or report;+share only the relevant manually redacted error and bounded observations.++With `metric=cpu` (or another explicit metric), the response has a `points`+array. Omitting `metric` instead returns a `metrics` object keyed by metric+name, not the same single-series shape. Inspect timestamps and `source` too:+`store` is persisted history; `memory` or `live` fallback does not establish+durable historical coverage. An HTTP 200 with no points proves neither zero+usage nor successful collection, and a non-zero current value does not prove+every historical interval is present.> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.-> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.+> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and+> `docker` are not current `resourceType` query values. Use the chart's current+> type rather than a display label or internal storage name.## Troubleshooting-- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists.-- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse.+- **Missing or stale history**: retain the affected panel, metric, range,+ collection time and current server/agent versions. Check the connection's+ last normal poll or agent report in **Settings → Infrastructure**, without+ running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw.+ Compare the panel's actual request, HTTP result, point timestamps and source.+ A valid connection test, another resource's graph or the existence of+ `metrics.db` does not establish this metric's collection or display.+- **Access or query error**: keep it separate from an empty result. Check the+ selected organisation, `monitoring:read` access, exact query type/ID and the+ requested range. A 402 is a history-range entitlement failure, not evidence+ that storage is empty; try an allowed range without buying another licence+ or resetting data.+- **Storage warning**: inspect a bounded existing startup/runtime error and+ the deployment's persistent mount and service-account access privately.+ Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+ for high writes or space pressure. Do not delete history, edit a database,+ shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.+- **Report still needed**: share the affected metric/range, versions,+ whether normal collection advances, and a manually redacted error; keep+ resource identities, response files and databases private. Follow+ [Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump.diff --git a/scripts/tests/test_setup_access_docs.py b/scripts/tests/test_setup_access_docs.pyindex d43f2dba75..834d49a1df 100644--- a/scripts/tests/test_setup_access_docs.py+++ b/scripts/tests/test_setup_access_docs.py@@ -17,9 +17,11 @@ import statimport subprocessimport tempfileimport unittest-from urllib.parse import parse_qs, urlsplit+from urllib.parse import parse_qs, quote, urlsplit-from test_api_auth_docs import ROOT, TEST_TOKEN, exercise_curl, recording_server+from test_api_auth_docs import (+ ROOT, TEST_TOKEN, exercise_curl, private_recording_server, request_helper,+)GUIDES = ("KUBERNETES", "METRICS_HISTORY", "CLOUD")@@ -186,14 +188,20 @@ class SetupAccessDocsTest(unittest.TestCase):class MetricsAccessDocsHTTPTest(unittest.TestCase):+ def history_request(self) -> str:+ # Use the same helper readers are told to define, not a fixture-only+ # implementation that could conceal a direct-curl response leak.+ return request_helper() + "\n" + recipe("METRICS_HISTORY", "pulse_api GET ")+def test_history_query_with_both_headers_and_hostile_curl_defaults(self):- request = recipe("METRICS_HISTORY", "curl ")+ request = self.history_request()expected_id = "pve 1:node/1:100&other=1"- request = request.replace("pve1:node1:100", expected_id)- with recording_server() as (port, records):+ request = request.replace("pve1%3Anode1%3A100", quote(expected_id, safe=""))+ with private_recording_server() as (port, records):for key, value in (("X-API-Token", TEST_TOKEN), ("Authorization", "Bearer " + TEST_TOKEN)):with self.subTest(header=key), tempfile.TemporaryDirectory() as temporary:- result = exercise_curl(self, Path(temporary), f"{key}: {value}", port, request)+ result = exercise_curl(self, Path(temporary), f"{key}: {value}", port,+ request, private_response=True)self.assertEqual(result.returncode, 0, result.stderr.decode())path, headers, method, body = records[-1]self.assertEqual(urlsplit(path).path, "/api/metrics-store/history")@@ -205,12 +213,57 @@ class MetricsAccessDocsHTTPTest(unittest.TestCase):def test_http_auth_license_and_server_errors_fail(self):for status in (401, 402, 403, 500):- with self.subTest(status=status), recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary:+ with self.subTest(status=status), private_recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary:result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port,- recipe("METRICS_HISTORY", "curl "))+ self.history_request(), private_response=True)self.assertEqual(result.returncode, 22, result.stderr.decode())self.assertEqual(len(records), 1)- self.assertIn(b'{"fixture":true}', result.stdout)+ self.assertIn(f"HTTP {status}".encode(), result.stdout)++ def test_redirect_and_incomplete_transport_are_not_empty_success(self):+ for status, partial, expected_exit in ((302, False, 1), (200, True, 18)):+ with self.subTest(status=status, partial=partial), private_recording_server(status, partial) as (port, records), tempfile.TemporaryDirectory() as temporary:+ result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port,+ self.history_request(), private_response=True)+ self.assertEqual(result.returncode, expected_exit, result.stderr.decode())+ self.assertEqual(len(records), 1)++ def test_repeated_reads_preserve_earlier_private_responses(self):+ with private_recording_server() as (port, records), tempfile.TemporaryDirectory() as temporary:+ home = Path(temporary)+ for _ in range(2):+ result = exercise_curl(self, home, f"X-API-Token: {TEST_TOKEN}", port,+ self.history_request(), private_response=True)+ self.assertEqual(result.returncode, 0, result.stderr.decode())+ self.assertEqual(len(records), 2)+ self.assertEqual(len(list((home / ".config/pulse").glob("api-response.*"))), 3)++ def test_query_types_match_current_handler_and_both_references(self):+ expected = {"node", "storage", "agent", "disk", "k8s", "vm", "system-container",+ "oci-container", "app-container", "docker-host"}+ handler = (ROOT / "internal/api/router.go").read_text().split(+ "func normalizeMetricsHistoryResourceType(", 1)[1].split("\n}\n", 1)[0]+ self.assertEqual(set(re.findall(r'case "([^"]+)":', handler)), expected)+ for name in ("API", "METRICS_HISTORY"):+ with self.subTest(guide=name):+ parameters = guide(name).split("### History" if name == "API" else+ "### History Query Parameters", 1)[1]+ types = parameters.split("- `resourceType` (required):", 1)[1].split(+ "- `resourceId`", 1)[0]+ self.assertEqual(set(re.findall(r"`([^`]+)`", types)), expected)++ def test_guidance_distinguishes_coverage_access_and_safe_collection(self):+ content = " ".join(guide("METRICS_HISTORY").split())+ for boundary in ("same Bash session", "not an installed Pulse command", "owner-only file",+ "not print or post it", "URL-encoding each query value", "no automatic retry",+ "a `points` array", "a `metrics` object", "timestamps and `source`",+ "does not establish durable historical coverage", "no points proves neither zero",+ "existence of `metrics.db` does not establish", "selected organisation",+ "not a repair for missing history", "does not migrate the existing history",+ "Do not delete history", "backup freeze/thaw", "manually redacted error"):+ self.assertIn(boundary, content)+ self.assertFalse(any("curl " in block for block in blocks("METRICS_HISTORY")),+ "history examples must use the existing private-response helper")if __name__ == "__main__":