Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Reconcile reliability roadmap and exit evidence

rcourtman3f59baa680fb
diff --git a/docs/release-control/control_plane.json b/docs/release-control/control_plane.json
index db8fa2da99..ea3b9021e8 100644
--- a/docs/release-control/control_plane.json
+++ b/docs/release-control/control_plane.json
@@ -98,7 +98,7 @@
"profile_id": "v6",
"kind": "maintenance",
"status": "active",
- "summary": "Priority set on 10 September 2026: stability, consistent updates and fixing reported issues are the highest current objective, to rebuild user trust through dependable delivery. Prioritize resolving release blockers and verifying qualification, publication and customer delivery through the existing release process. Keep failed attempts owned through repair and verified recovery without repeated founder chasing. Release consistency remains unconfirmed until actual release and recovery evidence establishes dependable end-to-end delivery. A repaired component, passing source check or started retry does not establish this outcome. Feature expansion is not a current priority.",
+ "summary": "Priority set on 10 September 2026: stability, consistent updates and fixing reported issues are the highest current objective, to rebuild user trust through dependable delivery. Prioritize resolving release blockers and verifying qualification, publication and customer delivery through the existing release process. Keep failed attempts owned through repair and verified recovery without repeated founder chasing. Release consistency remains unconfirmed until actual release and recovery evidence establishes dependable end-to-end delivery. A repaired component, passing source check or started retry does not establish this outcome. Feature expansion is not a current priority. Remaining work, reconciled delivered repairs and the existing manual exit evidence are recorded under L1 follow-up maintenance-release-delivery-and-observation and docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md. Keep this target active until actual release, installed recovery and ordinary operational observations support an explicit completion judgment. The planned recovery/change sequence remains deferred until that priority handoff.",
"completion_rule": "manual",
"proof_scope": "none"
}
diff --git a/docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md b/docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md
index 7162096142..c24785cc3f 100644
--- a/docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md
+++ b/docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md
@@ -68,7 +68,7 @@ The task timeline already has an accepted, bounded implementation contract in [B
- [Issue 2465](https://github.com/rcourtman/Pulse/issues/2465) records missed backup evaluation and a subsequent correction that retains existing alerts and exposes evaluation failure. Successful monitoring and visible failure remain prerequisites, not deferred feature work.
- [Issue 2587](https://github.com/rcourtman/Pulse/issues/2587) already has a PBS shortcut correction reported in v6.6.0-rc.1. Verify current behavior and delivery before scheduling more navigation work for it.
-Before implementing this roadmap, reconcile the task timeline ledger's stable-delivery condition for the 1815 correction with actual release contents. Current reliability work remains the active target. Trust defects found in matching, collection, evaluation, persistence or updates return to that work instead of being disguised as post-reliability enhancements.
+Reconciled on 18 October 2026: the #1815 backup-running correction `8c639846969c4f174aba874e7841df0ba0122de8` is included in published stable v6.5.0. Its stable-inclusion prerequisite is met. Current reliability work remains the active target, with remaining work and the manual exit evidence recorded in [the existing L1 delivery follow-up](records/maintenance-release-delivery-and-observation-2026-10-18.md). Trust defects found in matching, collection, evaluation, persistence or updates return to that work instead of being disguised as post-reliability enhancements.
## Priority and dependency order
@@ -162,7 +162,7 @@ The existing roadmap owns pickup of this sequence:
L15 and L19 keep their existing planned follow-ups and bounded-residual lane states. L20 governs any later expansion of executable actions. Demand statuses and signal counts are unchanged. The audit and roadmap direction are not additional independent customer signals.
-When reliability work is explicitly ready to yield priority, activate one bounded slice through the existing target and demand process. Reconcile the relevant ledger entry and contract, reserve its claim, and name its source owner and proof obligations before implementation. This tracked sequence is not evidence that the reliability target has completed and is not independent authority to start every item.
+When reliability work is explicitly ready to yield priority under [the recorded manual exit decision](records/maintenance-release-delivery-and-observation-2026-10-18.md#exit-decision-and-feature-pickup), activate one bounded slice through the existing target and demand process. Reconcile the relevant ledger entry and contract, reserve its claim, and name its source owner and proof obligations before implementation. This tracked sequence is not evidence that the reliability target has completed and is not independent authority to start every item.
## Browser audit evidence
diff --git a/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md b/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md
new file mode 100644
index 0000000000..9d083fef08
--- /dev/null
+++ b/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md
@@ -0,0 +1,42 @@
+# Reliability work and exit evidence
+
+Reconciled: 2026-10-18. Public source checked against Pulse `13c99c17c13659ecce920fd04b24a2617479cca8`. Release recovery observations are dated 2026-10-18 09:36 UTC. Later outcomes must replace these observations with their exact source and runtime evidence.
+
+The active target remains `v6-release-reliability`, with manual completion and `proof_scope=none`. Stability, consistent updates, reported defects and verified recovery retain priority. The existing L1 follow-up `maintenance-release-delivery-and-observation` owns this reconciliation and the release-delivery residual. The existing L6 notification gap owns destination delivery and recovery. This record adds no release gate, readiness assertion, target, lane or scheduling rule.
+
+## Reconciled completed work
+
+| Previous tracking | Reconciled evidence and limit |
+| --- | --- |
+| L1 said source-binding repair `f04c230` was still proposed as PR2091 and unreleased. | [PR2091](https://github.com/rcourtman/Pulse/pull/2091) merged on 15 September. Full commit `f04c230afbbf266238079afa065cfe50c6fd7226` is an ancestor of published stable [v6.5.0](https://github.com/rcourtman/Pulse/releases/tag/v6.5.0), peeled to `38c981a397c7a37f469b619200b5035bfc60b328`. It is no longer an unfinished source-landing item. Historical adverse CI evidence remains historical evidence, rather than a current failed check on that merged proposal. |
+| The backup task timeline waited for the stuck Backup Running correction to reach stable. | `8c639846969c4f174aba874e7841df0ba0122de8` is an ancestor of the same stable tag. [The #1815 reporter retest](https://github.com/rcourtman/Pulse/issues/1815#issuecomment-5851168092) confirmed the running-state correction and distinguished a failed secondary sync from a successful primary backup. The stable-inclusion prerequisite is met. The separate task-outcome opportunity remains under #1812 and L15, deferred behind reliability. |
+| The demand ledger still described #1753 identity and #1894 CPU regressions as awaiting their first successful retests. | [The #1753 reporter](https://github.com/rcourtman/Pulse/issues/1753#issuecomment-5662035612) reported no more rollups on v6.4.4-beta.3. That issue closed on 20 September. [The #1894 reporter](https://github.com/rcourtman/Pulse/issues/1894#issuecomment-5971141775) confirmed resolution before closure on 3 October. Mainline corrections `384dc53608eebca505b77119fdfc1af891edd6be` and `0f972f42f23a4b72dadc9bf98b026c9d70baee62` are in v6.5.0. Preserve their regression coverage without relisting these closed incidents as open reliability work. |
+| Native retry-prerequisite and publication-lock repairs were awaiting installed proof. | The existing recovery observation records the installed guard's waiting, ready and consumed states, the original automatic retry and actual publication, and installed concurrency/provenance proof for Infra PR767 and PR768. Retain that completed evidence. It does not establish the later disk-growth outcome or completion of the whole reliability target. |
+
+Ancestry was checked with `git merge-base --is-ancestor <full-commit> refs/tags/v6.5.0`. Preview or stable-line adaptation hashes are not interchangeable with mainline hashes. In particular, the #1753 `c1b671...` preview adaptation and #1894 `ac44f232...` v6.4.5 adaptation are not ancestors of v6.5.0. The mainline hashes above are the verified v6.5.0 authority.
+
+## Remaining work and acceptance evidence
+
+| Owning surface | Remaining outcome | Evidence required for disposition |
+| --- | --- | --- |
+| L1 release delivery and the existing maintenance source proposal | [PR2888](https://github.com/rcourtman/Pulse/pull/2888), head `09b82aec6dd30cc030abe89df125f6f399708fc2`, remains open. Its [Frontend job](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804) has three assertion failures. Its [macOS Intel job](https://github.com/rcourtman/Pulse/actions/runs/38660106183/job/116179086007) retains service-state/NAS test timeouts. | Diagnose the actual retained failures, preserve adverse evidence, pass required checks on the final source and land through normal review. Then identify the release containing the repair and qualify affected installed workflows. Local passes or unrelated green heads do not close those outcomes. |
+| Existing guest, agent, installer and monitoring owners, coordinated through L1 delivery | [Guest freeze/thaw #2439](https://github.com/rcourtman/Pulse/issues/2439), [Windows cooldown #2822](https://github.com/rcourtman/Pulse/issues/2822), [Windows disk collection #2619](https://github.com/rcourtman/Pulse/issues/2619), [Windows update #1820](https://github.com/rcourtman/Pulse/issues/1820) and [server memory #1803](https://github.com/rcourtman/Pulse/issues/1803) retain unverified customer outcomes. Current RC notes also retain unresolved guest-freeze and Windows-restart reports. | Bind each disposition to the affected source, installed version and workflow. For backup safety, independently observe thaw, fresh writes on every backed-up filesystem and workload liveness. For Windows, prove collection and service recovery after restart and the signed server-hosted update path. For memory, compare sustained steady usage and growth under the relevant workload. Source inclusion, an OK backup, a running VM or an initial render is insufficient. Any accepted residual needs an explicit scoped disposition and retained owner, without claiming field recovery. |
+| L6 `notification-delivery-terminal-failure-diagnosis` | [SMTP PR3045](https://github.com/rcourtman/Pulse/pull/3045) and [connectivity/DNS PR3049](https://github.com/rcourtman/Pulse/pull/3049) are landed. Real destination acceptance, release delivery, repeated-failure handling and destination recovery remain separate outcomes. [Issue #3047](https://github.com/rcourtman/Pulse/issues/3047) retains the misleading retries-exhausted label. | Prove controlled real-provider delivery and the affected release's installed path, distinguish terminal refusal from exhausted retries, and observe recovery after the destination is repaired. Preserve restart, queue, TLS and DNS failure evidence. Rolling fleet counters cannot attribute failures to these fixes or prove fleet recovery. See the existing [SMTP record](notification-smtp-delivery-reliability-2026-10-17.md) and [connectivity record](notification-connectivity-reliability-2026-10-18.md). |
+| L1 exact-candidate publication and observation | Latest stable is v6.5.0. [RC3](https://github.com/rcourtman/Pulse/releases/tag/v6.6.0-rc.3) published at 17 October 18:59:47 UTC from `524b10f54f23b6ffca71e73762fc2cfb926cae4f`. RC4 preparation `edf720ab9278a317ad881a9c9f375aa208ab90ce` is landed but unpublished at the observation time. | Verify the selected candidate's actual publication, source pair, assets and customer-facing update paths. Its own full 24-hour soak starts at actual product publication. RC3's minimum is 18 October 18:59:47 UTC. RC4's previous-publication spacing floor is that same time, which is not its own soak completion. No candidate inherits another candidate's elapsed soak. Stable must retain the fully soaked product source with release metadata only, pass the existing gates and have installed upgrade/recovery evidence. |
+| Existing release continuity and operational recovery owners, coordinated through L1 | [Scheduled Security Scan 39104703123](https://github.com/rcourtman/Pulse/actions/runs/39104703123) failed on HTTP/2 cancellation while reading publication inventory. The transport repair has accepted local review but protected delivery and the next genuine scheduled success remain unverified. Separately, [disk-footprint PR772](https://github.com/rcourtman/pulse-dev-infra/pull/772) is installed, with ordinary-run footprint and sustained free-space trend still pending. | Land and activate any required repair through normal gates, then observe the genuine scheduled recovery. Do not infer an orphaned tag or vulnerability from the interrupted inventory read. Measure ordinary publication footprints and storage trends across guest and backing host separately. A capacity increase, one healthy sample or source proof alone does not establish bounded growth. Retain known old-stable convergence debt separately from the successful exact RC3 convergence. |
+
+The dated release recovery snapshot is retained in workspace evidence at `.pulse-proof-artifacts/weekly-stable-cadence-20261014/weekly-release-monitor-progress.json`. At 09:36 UTC it records zero qualifying new daily betas out of three, no first verified weekly stable under the 14 October cadence policy, RC4 unpublished with its own soak unstarted, and disk-footprint observation incomplete. Its older embedded observations must not override newer source landing or installed acceptance. In particular, Pro PR478 was ledger-only and has merged, and the licence-only deployment already has its own verified installation receipt. Neither is unfinished source work in this residual.
+
+Two observation residuals remain with their existing release owner. RC4's exact request-origin explanation is unreconciled, without authority to replay or issue a release request. Separately, [scheduled convergence reconciliation 39112622024](https://github.com/rcourtman/Pulse/actions/runs/39112622024) retains old-stable debt because the original workflow owner run is unavailable. Its disposition and expected normal reconciliation need evidence. [Exact RC3 convergence 39071952106](https://github.com/rcourtman/Pulse/actions/runs/39071952106) passed, so the historical debt must not be relabelled an RC3 convergence failure.
+
+## Exit decision and feature pickup
+
+The existing manual target decision must review these outcomes together:
+
+1. Relevant maintenance repairs have final-source qualification, normal landing, release inclusion and affected installed-workflow evidence. High-impact unresolved incidents have an evidenced resolution or an explicit owned residual that does not contradict the reliability outcome being claimed.
+2. The selected product candidate completes its own actual-publication-based 24-hour soak. Stable publication preserves that product source, passes the existing qualification and promotion gates, and verifies assets, customer delivery and installed upgrade/recovery.
+3. Preserve the existing recovery observation commitment: three consecutive new daily betas without manual release rescue and the first verified weekly stable under the 14 October cadence policy. The excluded beta.3 baseline receives no credit. Ordinary product commits, legitimate RC windows and no-change skips do not become release interventions or reset the count. Accepting an RC in place of the next beta does not silently turn an RC into a counted beta. This evidence commitment does not create another release gate or make a count alone sufficient to complete the target.
+4. Required operational repairs have matching installed evidence and observed normal recovery. The scheduled scan and ordinary publisher storage outcomes above remain owned until their actual results are reconciled. Retain already verified retry-guard, lock-scope and licence-only installation evidence rather than reopening them.
+5. Reconcile the dated observations and any new material regressions, record the manual reliability completion judgment, then explicitly change the active target before starting feature expansion. Green historical assertions or release gates alone do not make that judgment.
+
+The [recovery and change investigation plan](../RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md) remains under the existing planned `v6-product-lane-expansion` target and L15/L19 follow-ups. After reliability explicitly yields priority, select one bounded slice through the existing demand, contract, ownership and claim process. Its backup-running stable-inclusion prerequisite is already met. Planned feature residuals across the wider lane map do not all have to be implemented to finish this reliability target.
diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.json
index 67b085e96d..659b69a0de 100644
--- a/docs/release-control/v6/internal/status.json
+++ b/docs/release-control/v6/internal/status.json
@@ -5102,7 +5102,7 @@
"status": "partial",
"completion": {
"state": "bounded-residual",
- "summary": "Self-hosted release confidence reached its target and was re-proved for v6.3.1 with exact-SHA dry-run, stable activation, published-asset integrity, installer, demo, container, Helm, private paid-runtime convergence, and bounded recovery from GitHub activation-marker propagation. Current maintenance residual: f04c230 source-binding repair is proposed as PR2091, not landed or released; its WebSocket/benchmark CI failures remain owned. Exact RC scope, qualification and required observation coverage remain outstanding.",
+ "summary": "Self-hosted release confidence retains its historical v6.3.1 qualification evidence. Reconciled 18 October 2026: source-binding repair f04c230 landed through PR2091 and is included in stable v6.5.0. Remaining reliability work is final-source maintenance qualification, affected customer workflow recovery, exact-candidate publication and its own full soak, stable installed upgrade/recovery, and observed scheduled-scan and publisher storage recovery. The existing maintenance-release-delivery-and-observation follow-up and docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md own the current evidence and manual exit decision. No lane closure or priority handoff is claimed.",
"tracking": [
{
"kind": "lane-followup",
@@ -5130,6 +5130,12 @@
"path": "docs/release-control/v6/internal/records/known-rc-issue-closure-for-ga-installer-bundle-fallback-2026-05-01.md",
"kind": "file"
},
+ {
+ "repo": "pulse",
+ "path": "docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md",
+ "kind": "file",
+ "evidence_tier": "test-proof"
+ },
{
"repo": "pulse",
"path": "docs/release-control/v6/internal/records/rc-to-ga-promotion-readiness-rehearsal-2026-08-22.md",
@@ -9470,7 +9476,7 @@
},
{
"id": "storage-recovery-post-rc-hardening",
- "summary": "Track broader storage health UX polish and recovery timeline hardening beyond the current RC floor, including additional filter coherence edge cases and storage panel presentation improvements. Post-reliability recovery priorities are orders 1, 3 and 5 in docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md: cross-system backup task outcomes under docs/BACKUP_TASK_TIMELINE_CONTRACT.md, one understandable workload protection assessment, then demand-dependent retained recovery-test evidence. Pick up through planned target v6-product-lane-expansion only when v6-release-reliability explicitly yields priority. Reconcile the existing task-timeline demand entry and stable-delivery prerequisite before activation. Recovery-test execution remains a separate L20 admission decision.",
+ "summary": "Track broader storage health UX polish and recovery timeline hardening beyond the current RC floor, including additional filter coherence edge cases and storage panel presentation improvements. Post-reliability recovery priorities are orders 1, 3 and 5 in docs/release-control/v6/internal/RECOVERY_AND_CHANGE_INVESTIGATION_PLAN.md: cross-system backup task outcomes under docs/BACKUP_TASK_TIMELINE_CONTRACT.md, one understandable workload protection assessment, then demand-dependent retained recovery-test evidence. Pick up through planned target v6-product-lane-expansion only when v6-release-reliability explicitly yields priority. The #1815 backup-running correction is verified in stable v6.5.0, so that inclusion prerequisite is met. Preserve its regressions and reconcile any new evidence before activation. The remaining priority prerequisite is the explicit reliability handoff recorded by maintenance-release-delivery-and-observation. Recovery-test execution remains a separate L20 admission decision.",
"owner": "project-owner",
"status": "planned",
"recorded_at": "2026-03-14",
@@ -9666,7 +9672,7 @@
},
{
"id": "maintenance-release-delivery-and-observation",
- "summary": "Land the stable source-binding repair, diagnose retained CI failures, qualify a coherent maintenance RC, verify installed upgrade/recovery, and reconcile exact-candidate observation coverage without changing stable services.",
+ "summary": "Reconcile final-source maintenance qualification and affected installed customer outcomes, selected-candidate publication and its own 24-hour soak, stable promotion and installed upgrade/recovery, the existing unassisted-release observation commitment, and actual scheduled-scan and publisher storage recovery. Delivered PR2091 and the stable #1815 correction are reconciled rather than unfinished landing work. Current dated evidence and manual target exit criteria are in docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md. Keep v6-release-reliability active until those outcomes support an explicit completion judgment.",
"owner": "pulse-maintainer",
"status": "planned",
"recorded_at": "2026-09-15",