Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain container CPU scales and passive comparisons
pulse-triage[bot]0c283d2dd561
diff --git a/docs/DOCKER.md b/docs/DOCKER.mdindex ac46ce3f9e..e35f9f2cf1 100644--- a/docs/DOCKER.md+++ b/docs/DOCKER.md@@ -123,6 +123,57 @@ services:---+## Container CPU readings++Pulse shows container CPU as a share of the **Docker/Podman host's total CPU+capacity**: 100% means all its logical CPUs. `docker stats` and `podman stats`+normally use **100% for one logical CPU**, so their readings can exceed 100%.+For that scale, divide the stats percentage by the CPU count reported for the+runtime host to compare it with Pulse.++If the engine runs inside a VM, use that VM's logical CPUs, not the physical+Proxmox host's CPUs. Pulse's percentage is not utilisation of the container's+own CPU quota or cpuset. If the reported CPU count differs from your configured+VM size, retain both in the comparison rather than guessing a divisor.++| Example host logical CPUs | Stats CPU | Pulse CPU before display rounding |+| --- | --- | --- |+| 4 | 240% | 60% |+| 6 | 16.52% | About 2.75% |+| 6 | 1.04% | About 0.17% |++These are examples, not an assumption about your host. A smaller Pulse value+alone does not establish a collection fault. Whole-percent table labels,+including those in **v6.5.0-rc.1**, can display a fractional reading below 0.5%+as 0%. Use **CPU History** rather than the rounded label; display rounding does+not change stored readings or the values used by CPU alert evaluation. A dash+or missing History is unavailable data, not a measured zero.++For a useful comparison:++1. Match the same container on the same runtime host. Compare a similar time+ interval in CPU History with `docker stats` or `podman stats`; separate+ sampling intervals can legitimately differ.+2. Apply the reported host CPU count only to a stats reading on the per-CPU+ scale. Do not divide Pulse History again, or compare with an average since+ container start as though it were a recent interval.+3. Check that observation times and History points advance during ordinary+ monitoring. A connection marked Active does not prove fresh CPU samples.+4. If the normalised readings still disagree, or History stops advancing,+ report the server and agent versions, reported CPU count, sample times and+ bounded readings. Do not restart workloads, create CPU load, lower alert+ thresholds or export diagnostics just to make this comparison.++CPU alerts use the same host-capacity scale, with the configured thresholds,+duration and suppression rules. Non-zero History proves a collected reading,+not that an alert should fire or that its notification was delivered. If a+normal workload already demonstrates an alert problem, retain the reading,+threshold, duration and observed result; no forced alert test is needed. See+[safe performance measurements](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+when measuring the Pulse server itself rather than a monitored container.++---+## 🔄 UpdatesThese steps update the Pulse server, not the containers it monitors. Record thediff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex dd9e0a54ef..213b11b17d 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -251,6 +251,20 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.follow the [platform-specific guide](TEMPERATURE_MONITORING.md) for any neededsetup change during a maintenance window.+#### Container CPU differs from Docker or Podman stats++Pulse uses the runtime host's total CPU capacity; `docker stats` and+`podman stats` normally use 100% per logical CPU. Divide a per-CPU stats reading+by the **reported runtime host CPU count**, not the physical Proxmox host's+count when the engine runs inside a VM. Compare the same container and time+interval in **CPU History**, not just a rounded table label. A dash or missing+History is unavailable, not zero; an Active connection alone is not fresh CPU+evidence.++See [container CPU readings](DOCKER.md#container-cpu-readings) for worked+examples, rounding and alert limits, and a passive comparison. Do not restart+workloads, create CPU load or lower alert thresholds to test this.+#### Docker hosts appearing/disappearingCloned hosts can share a **saved Pulse agent ID**, not just an OS machine ID.diff --git a/frontend-modern/browser-tests/container-cpu-help.cjs b/frontend-modern/browser-tests/container-cpu-help.cjsnew file mode 100644index 0000000000..dbaa300a2d--- /dev/null+++ b/frontend-modern/browser-tests/container-cpu-help.cjs@@ -0,0 +1,117 @@+// The real Docs renderer, router and shipped help. No runtime socket, workload,+// credentials, alert test or diagnostics collection is exercised.+const assert = require('node:assert/strict');+const fs = require('node:fs');+const path = require('node:path');+const crypto = require('node:crypto');+const { chromium, webkit } = require('playwright');++(async () => {+ const root = path.resolve('frontend-modern');+ const artifacts = path.join(root, 'node_modules', 'container-cpu-help-browser');+ fs.mkdirSync(artifacts, { recursive: true });+ process.chdir(root);+ const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));+ const server = await createServer({+ root, configFile: path.join(root, 'vite.config.ts'),+ cacheDir: path.join(root, 'node_modules', '.vite-container-cpu-help'),+ server: { host: '127.0.0.1', port: 5265, strictPort: true },+ optimizeDeps: { noDiscovery: true, include: [] },+ });+ const results = [];+ let browser;+ try {+ await server.listen();+ for (const [engine, width, tone] of [+ ['chromium', 1280, 'light'], ['chromium', 1280, 'dark'],+ ['webkit', 390, 'light'], ['webkit', 390, 'dark'],+ ]) {+ browser = await { chromium, webkit }[engine].launch(engine === 'chromium'+ ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }+ : { headless: true });+ const page = await browser.newPage({ viewport: { width, height: 900 },+ ...(engine === 'webkit' ? { isMobile: true, hasTouch: true } : {}) });+ const errors = [];+ const operations = [];+ page.on('pageerror', (error) => errors.push(error.message));+ page.on('console', (message) => {+ if (message.type() === 'error') errors.push(message.text());+ });+ page.on('request', (request) => {+ if (request.url().includes('/api/') || request.method() !== 'GET')+ operations.push({ method: request.method(), url: request.url() });+ });+ await page.goto('http://127.0.0.1:5265/browser-tests/docs-fragment-navigation.html?scenario=troubleshooting',+ { waitUntil: 'domcontentloaded', timeout: 60_000 });+ if (tone === 'dark') await page.evaluate(async () => {+ document.documentElement.classList.add('dark');+ await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)));+ });+ const entry = page.getByRole('heading', {+ name: 'Container CPU differs from Docker or Podman stats', exact: true,+ });+ await entry.waitFor();+ await entry.evaluate((heading) => heading.scrollIntoView({ block: 'start' }));+ await page.screenshot({ path: path.join(artifacts, `${engine}-${tone}-entry.png`) });+ const guideLink = page.getByRole('link', { name: 'container CPU readings', exact: true });+ if (engine === 'webkit') await guideLink.tap();+ else {+ await guideLink.focus();+ await page.keyboard.press('Enter');+ }+ const heading = page.getByRole('heading', { name: 'Container CPU readings', exact: true });+ await heading.waitFor();+ await page.waitForFunction(() => {+ const h = document.getElementById('container-cpu-readings');+ return h && h.getBoundingClientRect().top >= 0 && h.getBoundingClientRect().top < innerHeight;+ });+ assert.ok(page.url().endsWith('/docs/DOCKER#container-cpu-readings'));+ const article = (await page.locator('article').innerText()).replace(/\s+/g, ' ');+ for (const phrase of ["Docker/Podman host's total CPU capacity", '100% for one logical CPU',+ 'not an assumption about your host', 'v6.5.0-rc.1', 'unavailable data, not a measured zero',+ 'Do not divide Pulse History again', 'Active does not prove fresh CPU samples',+ 'not that an alert should fire or that its notification was delivered',+ 'Do not restart workloads, create CPU load, lower alert thresholds'])+ assert.ok(article.includes(phrase), phrase);+ const table = page.locator('article table').filter({ hasText: 'Example host logical CPUs' });+ assert.equal(await table.getByRole('row').count(), 4);+ assert.equal(await table.getByRole('columnheader').count(), 3);+ const rows = await table.getByRole('row').allTextContents();+ assert.ok(rows[1].includes('240%') && rows[1].includes('60%'));+ assert.ok(rows[2].includes('16.52%') && rows[2].includes('2.75%'));+ assert.ok(rows[3].includes('1.04%') && rows[3].includes('0.17%'));+ assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));+ await table.scrollIntoViewIfNeeded();+ await page.screenshot({ path: path.join(artifacts, `${engine}-${tone}-examples.png`) });+ const performance = page.getByRole('link', { name: 'safe performance measurements', exact: true });+ if (engine === 'webkit') await performance.tap();+ else {+ await performance.focus();+ await page.keyboard.press('Enter');+ }+ await page.getByRole('heading', { name: 'Excessive CPU, writes or database growth', exact: true }).waitFor();+ await page.waitForFunction(() => {+ const h = document.getElementById('excessive-cpu-writes-or-database-growth');+ return h && h.getBoundingClientRect().top >= 0 && h.getBoundingClientRect().top < innerHeight;+ });+ assert.ok(page.url().endsWith('/docs/TROUBLESHOOTING#excessive-cpu-writes-or-database-growth'));+ assert.deepEqual(errors, []);+ assert.deepEqual(operations, []);+ results.push({ engine, version: browser.version(), width, tone, exampleRows: rows,+ entryLink: engine === 'webkit' ? 'touch' : 'keyboard', performanceLink: true,+ noDocumentOverflow: true, errors, operations });+ await browser.close();+ browser = undefined;+ }+ const docs = ['DOCKER.md', 'TROUBLESHOOTING.md'].map((name) => ({ name,+ sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(root, 'public', 'docs', name))).digest('hex'),+ }));+ const result = { result: 'passed', playwright: require('playwright/package.json').version,+ scope: 'shipped CPU help rendering and navigation, not collector accuracy or alert delivery', docs, results };+ fs.writeFileSync(path.join(artifacts, 'result.json'), JSON.stringify(result, null, 2) + '\n');+ console.log(JSON.stringify(result));+ } finally {+ if (browser) await browser.close();+ await server.close();+ }+})().catch((error) => { console.error(error); process.exitCode = 1; });diff --git a/frontend-modern/public/docs/DOCKER.md b/frontend-modern/public/docs/DOCKER.mdindex ac46ce3f9e..e35f9f2cf1 100644--- a/frontend-modern/public/docs/DOCKER.md+++ b/frontend-modern/public/docs/DOCKER.md@@ -123,6 +123,57 @@ services:---+## Container CPU readings++Pulse shows container CPU as a share of the **Docker/Podman host's total CPU+capacity**: 100% means all its logical CPUs. `docker stats` and `podman stats`+normally use **100% for one logical CPU**, so their readings can exceed 100%.+For that scale, divide the stats percentage by the CPU count reported for the+runtime host to compare it with Pulse.++If the engine runs inside a VM, use that VM's logical CPUs, not the physical+Proxmox host's CPUs. Pulse's percentage is not utilisation of the container's+own CPU quota or cpuset. If the reported CPU count differs from your configured+VM size, retain both in the comparison rather than guessing a divisor.++| Example host logical CPUs | Stats CPU | Pulse CPU before display rounding |+| --- | --- | --- |+| 4 | 240% | 60% |+| 6 | 16.52% | About 2.75% |+| 6 | 1.04% | About 0.17% |++These are examples, not an assumption about your host. A smaller Pulse value+alone does not establish a collection fault. Whole-percent table labels,+including those in **v6.5.0-rc.1**, can display a fractional reading below 0.5%+as 0%. Use **CPU History** rather than the rounded label; display rounding does+not change stored readings or the values used by CPU alert evaluation. A dash+or missing History is unavailable data, not a measured zero.++For a useful comparison:++1. Match the same container on the same runtime host. Compare a similar time+ interval in CPU History with `docker stats` or `podman stats`; separate+ sampling intervals can legitimately differ.+2. Apply the reported host CPU count only to a stats reading on the per-CPU+ scale. Do not divide Pulse History again, or compare with an average since+ container start as though it were a recent interval.+3. Check that observation times and History points advance during ordinary+ monitoring. A connection marked Active does not prove fresh CPU samples.+4. If the normalised readings still disagree, or History stops advancing,+ report the server and agent versions, reported CPU count, sample times and+ bounded readings. Do not restart workloads, create CPU load, lower alert+ thresholds or export diagnostics just to make this comparison.++CPU alerts use the same host-capacity scale, with the configured thresholds,+duration and suppression rules. Non-zero History proves a collected reading,+not that an alert should fire or that its notification was delivered. If a+normal workload already demonstrates an alert problem, retain the reading,+threshold, duration and observed result; no forced alert test is needed. See+[safe performance measurements](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)+when measuring the Pulse server itself rather than a monitored container.++---+## 🔄 UpdatesThese steps update the Pulse server, not the containers it monitors. Record thediff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex dd9e0a54ef..213b11b17d 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -251,6 +251,20 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.follow the [platform-specific guide](TEMPERATURE_MONITORING.md) for any neededsetup change during a maintenance window.+#### Container CPU differs from Docker or Podman stats++Pulse uses the runtime host's total CPU capacity; `docker stats` and+`podman stats` normally use 100% per logical CPU. Divide a per-CPU stats reading+by the **reported runtime host CPU count**, not the physical Proxmox host's+count when the engine runs inside a VM. Compare the same container and time+interval in **CPU History**, not just a rounded table label. A dash or missing+History is unavailable, not zero; an Active connection alone is not fresh CPU+evidence.++See [container CPU readings](DOCKER.md#container-cpu-readings) for worked+examples, rounding and alert limits, and a passive comparison. Do not restart+workloads, create CPU load or lower alert thresholds to test this.+#### Docker hosts appearing/disappearingCloned hosts can share a **saved Pulse agent ID**, not just an OS machine ID.diff --git a/frontend-modern/src/utils/__tests__/docsLinks.test.ts b/frontend-modern/src/utils/__tests__/docsLinks.test.tsindex c5d89ea99b..0ccae13f89 100644--- a/frontend-modern/src/utils/__tests__/docsLinks.test.ts+++ b/frontend-modern/src/utils/__tests__/docsLinks.test.ts@@ -480,6 +480,40 @@ describe('docsLinks', () => {);});+ it('explains container CPU scales without inventing a zero or forcing an alert', () => {+ const docker = readFileSync(path.join(repoRoot, 'docs', 'DOCKER.md'), 'utf8');+ const cpu = docker+ .split('## Container CPU readings\n')[1]+ .split('\n---')[0]+ .replace(/\s+/g, ' ');+ expect(cpu).toContain("Docker/Podman host's total CPU capacity");+ expect(cpu).toContain('100% for one logical CPU');+ expect(cpu).toContain('CPU count reported for the runtime host');+ expect(cpu).toContain("not the physical Proxmox host's CPUs");+ expect(cpu).toContain("not utilisation of the container's own CPU quota or cpuset");+ expect(cpu).toContain('not an assumption about your host');+ expect(cpu).toContain('v6.5.0-rc.1');+ expect(cpu).toContain('not change stored readings or the values used by CPU alert evaluation');+ expect(cpu).toContain('unavailable data, not a measured zero');+ expect(cpu).toContain('Do not divide Pulse History again');+ expect(cpu).toContain('separate sampling intervals can legitimately differ');+ expect(cpu).toContain('Active does not prove fresh CPU samples');+ expect(cpu).toContain('not that an alert should fire or that its notification was delivered');+ expect(cpu).toContain('Do not restart workloads, create CPU load, lower alert thresholds');+ expect(cpu).toContain('no forced alert test is needed');++ const troubleshooting = readFileSync(path.join(repoRoot, 'docs', 'TROUBLESHOOTING.md'), 'utf8');+ const entry = troubleshooting+ .split('#### Container CPU differs from Docker or Podman stats\n')[1]+ .split('\n#### ')[0]+ .replace(/\s+/g, ' ');+ expect(entry).toContain('DOCKER.md#container-cpu-readings');+ expect(entry).toContain('reported runtime host CPU count');+ expect(entry).toContain('same container and time interval');+ expect(entry).toContain('unavailable, not zero');+ expect(entry).toContain('Do not restart workloads, create CPU load or lower alert thresholds');+ });+it('keeps clone identity recovery separate from destructive OS or credential resets', () => {const troubleshooting = readFileSync(path.join(repoRoot, 'docs', 'TROUBLESHOOTING.md'), 'utf8');const agentGuide = readFileSync(path.join(repoRoot, 'docs', 'UNIFIED_AGENT.md'), 'utf8');diff --git a/scripts/installtests/container_cpu_guidance_test.go b/scripts/installtests/container_cpu_guidance_test.gonew file mode 100644index 0000000000..9cbbe8aa93--- /dev/null+++ b/scripts/installtests/container_cpu_guidance_test.go@@ -0,0 +1,52 @@+package installtests++import (+ "math"+ "os"+ "regexp"+ "strconv"+ "testing"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+)++// Check the guide's actual worked examples against the normaliser used by+// container ingestion, History and alert evaluation, not a second formula.+func TestContainerCPUGuidanceExamplesMatchNormalisation(t *testing.T) {+ guide, err := os.ReadFile(repoFile("docs", "DOCKER.md"))+ if err != nil {+ t.Fatal(err)+ }+ rows := regexp.MustCompile(`(?m)^\| (\d+) \| ([\d.]+)% \| (?:About )?([\d.]+)% \|$`).FindAllStringSubmatch(string(guide), -1)+ if len(rows) != 3 {+ t.Fatalf("expected three worked host-capacity CPU examples, got %d", len(rows))+ }+ for _, row := range rows {+ cpus, err := strconv.Atoi(row[1])+ if err != nil {+ t.Fatal(err)+ }+ raw, err := strconv.ParseFloat(row[2], 64)+ if err != nil {+ t.Fatal(err)+ }+ printed, err := strconv.ParseFloat(row[3], 64)+ if err != nil {+ t.Fatal(err)+ }+ actual := models.NormalizeDockerContainerCPUCapacityPercent(raw, cpus)+ if math.Abs(actual-printed) > 0.005 {+ t.Errorf("%s disagrees with the production normaliser: %g", row[0], actual)+ }+ }+ for _, name := range []string{"DOCKER.md", "TROUBLESHOOTING.md"} {+ doc, err := os.ReadFile(repoFile("docs", name))+ if err != nil {+ t.Fatal(err)+ }+ shipped, err := os.ReadFile(repoFile("frontend-modern", "public", "docs", name))+ if err != nil || string(doc) != string(shipped) {+ t.Fatalf("%s shipped help does not match the checked canonical guide", name)+ }+ }+}