Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Respect construction and shared session login lifetimes

rcourtman6bf9e818e8c2
diff --git a/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md b/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md
index 9d083fef08..9fd9ced612 100644
--- a/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md
+++ b/docs/release-control/v6/internal/records/maintenance-release-delivery-and-observation-2026-10-18.md
@@ -19,7 +19,7 @@ Ancestry was checked with `git merge-base --is-ancestor <full-commit> refs/tags/
| Owning surface | Remaining outcome | Evidence required for disposition |
| --- | --- | --- |
-| L1 release delivery and the existing maintenance source proposal | [PR2888](https://github.com/rcourtman/Pulse/pull/2888), head `09b82aec6dd30cc030abe89df125f6f399708fc2`, remains open. Its [Frontend job](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804) has three assertion failures. Its [macOS Intel job](https://github.com/rcourtman/Pulse/actions/runs/38660106183/job/116179086007) retains service-state/NAS test timeouts. | Diagnose the actual retained failures, preserve adverse evidence, pass required checks on the final source and land through normal review. Then identify the release containing the repair and qualify affected installed workflows. Local passes or unrelated green heads do not close those outcomes. |
+| L1 release delivery and the existing maintenance source proposal | [PR2888](https://github.com/rcourtman/Pulse/pull/2888) remains open under source qualification. The old `09b82a...` Frontend and native failures are retained below. Integrated head `4b86af9a330d555cde0972bf1759b07bbaa94d64` exposed a cumulative NAS fixture alarm and a real conflict between caller-owned construction and shared session renewal. The per-control fixture runner is in `0b3261e1aa1356f291030622a91dca244324cd0b`. The login ownership correction is undergoing qualification. | Preserve the actual failed evidence, pass required checks on the final source and land through normal review. Then identify the release containing the repair and qualify affected installed workflows. Worker passes or unrelated green heads do not close those outcomes. |
| Existing guest, agent, installer and monitoring owners, coordinated through L1 delivery | [Guest freeze/thaw #2439](https://github.com/rcourtman/Pulse/issues/2439), [Windows cooldown #2822](https://github.com/rcourtman/Pulse/issues/2822), [Windows disk collection #2619](https://github.com/rcourtman/Pulse/issues/2619), [Windows update #1820](https://github.com/rcourtman/Pulse/issues/1820) and [server memory #1803](https://github.com/rcourtman/Pulse/issues/1803) retain unverified customer outcomes. Current RC notes also retain unresolved guest-freeze and Windows-restart reports. | Bind each disposition to the affected source, installed version and workflow. For backup safety, independently observe thaw, fresh writes on every backed-up filesystem and workload liveness. For Windows, prove collection and service recovery after restart and the signed server-hosted update path. For memory, compare sustained steady usage and growth under the relevant workload. Source inclusion, an OK backup, a running VM or an initial render is insufficient. Any accepted residual needs an explicit scoped disposition and retained owner, without claiming field recovery. |
| L6 `notification-delivery-terminal-failure-diagnosis` | [SMTP PR3045](https://github.com/rcourtman/Pulse/pull/3045) and [connectivity/DNS PR3049](https://github.com/rcourtman/Pulse/pull/3049) are landed. Real destination acceptance, release delivery, repeated-failure handling and destination recovery remain separate outcomes. [Issue #3047](https://github.com/rcourtman/Pulse/issues/3047) retains the misleading retries-exhausted label. | Prove controlled real-provider delivery and the affected release's installed path, distinguish terminal refusal from exhausted retries, and observe recovery after the destination is repaired. Preserve restart, queue, TLS and DNS failure evidence. Rolling fleet counters cannot attribute failures to these fixes or prove fleet recovery. See the existing [SMTP record](notification-smtp-delivery-reliability-2026-10-17.md) and [connectivity record](notification-connectivity-reliability-2026-10-18.md). |
| L1 exact-candidate publication and observation | Latest stable is v6.5.0. [RC3](https://github.com/rcourtman/Pulse/releases/tag/v6.6.0-rc.3) published at 17 October 18:59:47 UTC from `524b10f54f23b6ffca71e73762fc2cfb926cae4f`. RC4 preparation `edf720ab9278a317ad881a9c9f375aa208ab90ce` is landed but unpublished at the observation time. | Verify the selected candidate's actual publication, source pair, assets and customer-facing update paths. Its own full 24-hour soak starts at actual product publication. RC3's minimum is 18 October 18:59:47 UTC. RC4's previous-publication spacing floor is that same time, which is not its own soak completion. No candidate inherits another candidate's elapsed soak. Stable must retain the fully soaked product source with release metadata only, pass the existing gates and have installed upgrade/recovery evidence. |
@@ -29,6 +29,16 @@ The dated release recovery snapshot is retained in workspace evidence at `.pulse
Two observation residuals remain with their existing release owner. RC4's exact request-origin explanation is unreconciled, without authority to replay or issue a release request. Separately, [scheduled convergence reconciliation 39112622024](https://github.com/rcourtman/Pulse/actions/runs/39112622024) retains old-stable debt because the original workflow owner run is unavailable. Its disposition and expected normal reconciliation need evidence. [Exact RC3 convergence 39071952106](https://github.com/rcourtman/Pulse/actions/runs/39071952106) passed, so the historical debt must not be relabelled an RC3 convergence failure.
+## Maintenance source qualification in progress
+
+The initial [Frontend failure](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804) consists of three document-structure assertions. The correction asserts named workflows, required destinations and shipped headings, and scopes the opening safety link to its intended paragraph. It preserves command and safety assertions while allowing additional troubleshooting content.
+
+The [original Intel timeout](https://github.com/rcourtman/Pulse/actions/runs/38660106183/job/116179086007) and [integrated Intel timeout](https://github.com/rcourtman/Pulse/actions/runs/39135739613/job/117699764462) retain adverse evidence. The latter completed fifteen NAS controls before exhausting a two-minute alarm shared across 23 controls. The selected isolated Python interpreter and a discovered per-control runner address fixture execution and aggregate budgets. All 59 NAS, TrueNAS, service-state and log-sink controls passed worker proof, together with rejection of skipped, missing and failed controls. Actual installer operation deadlines and the native phase's ten-minute alarm are unchanged. Fresh native CI is still required.
+
+The integrated source's [backend partition 0](https://github.com/rcourtman/Pulse/actions/runs/39135739571/job/117701554206) and [partition 1](https://github.com/rcourtman/Pulse/actions/runs/39135739571/job/117701554230) failures were reproduced on the worker at `0b3261e1aa1356f291030622a91dca244324cd0b`. Constructor cancellation returned promptly but left its password request active. The integrated constructor had joined the detached login intended for concurrent readers renewing an accepted session. Initial authentication now owns a synchronous bounded JSON/form exchange. Shared renewal retains its separate caller-independent lifetime, generation coalescing and timeout. Existing constructor, startup and cluster controls must pass alongside shared-renewal cancellation and new client-timeout controls before source qualification closes.
+
+These are source repairs and qualification evidence. Release inclusion, affected installed workflows and the manual reliability exit decision remain open. No target, lane state or priority changes with this work.
+
## Exit decision and feature pickup
The existing manual target decision must review these outcomes together:
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index dea75ac8a1..489ed526de 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -232,6 +232,14 @@ reads use their own contexts, not the ended construction context. TLS pins,
CA verification, redirects, first-use admission, auth encodings, retry cadence,
provider permissions, guest/backup fences and History are unchanged.
+Initial PVE login runs synchronously under the caller and the configured total
+login timeout, including JSON/form fallback. It never joins the detached renewal
+used by an accepted session. Later expiry/401 renewal still coalesces by session
+generation under the client's bounded timeout, so cancellation of its starter or
+one waiting read cannot abandon other active readers. Both paths share the same
+password exchange and session parser. Ending an unaccepted constructor now also
+ends its password request, without changing the rendered health outcome.
+
`TestClientConstructionHonoursContext` in both client packages covers ended
password/token construction, in-flight JSON/form cancellation/deadlines,
independent later reads and wrong-pin refusal. Cluster caller-budget controls
@@ -243,6 +251,11 @@ server healthy, and accepted construction alone still cannot do so. These are
secret-free local fixtures, not installed recovery, native guest safety or an
attribution of any reporter's failure.
+`TestClientConstructionHonoursClientTimeout` additionally exercises both login
+encodings under the PVE client's own timeout while its caller remains live.
+Existing shared-renewal starter/waiter cancellation, expiry, late-401, failure
+recovery and stalled-login controls retain the accepted-session boundary.
+
### Cluster recovery requires a usable polling session
diff --git a/pkg/proxmox/client.go b/pkg/proxmox/client.go
index 680f5ba141..eabbed51fa 100644
--- a/pkg/proxmox/client.go
+++ b/pkg/proxmox/client.go
@@ -437,21 +437,32 @@ func NewClientWithContext(ctx context.Context, cfg ClientConfig) (*Client, error
return client, nil
}
-// authenticate performs password-based authentication
+// Initial authentication belongs only to the constructing caller. It has no
+// retained session or other readers whose login must outlive that caller.
func (c *Client) authenticate(ctx context.Context) error {
- return c.renewAuthentication(ctx, nil)
+ loginCtx, cancel := context.WithTimeout(ctx, c.sessionRenewalTimeout())
+ defer cancel()
+ auth, _ := c.authSnapshot()
+ return c.authenticatePassword(loginCtx, auth.user+"@"+auth.realm)
+}
+
+func (c *Client) authenticatePassword(ctx context.Context, username string) error {
+ err := c.authenticateJSON(ctx, username, c.config.Password)
+ if shouldFallbackToForm(err) {
+ err = c.authenticateForm(ctx, username, c.config.Password)
+ }
+ return err
}
// expected binds expiry or a 401 to the session actually used by that read.
// A late rejection of an old ticket cannot trigger another renewal after a
-// concurrent request has already replaced it. Explicit authentication (nil)
-// still owns the existing JSON/form fallback and failed-attempt behaviour.
-func (c *Client) renewAuthentication(ctx context.Context, expected *uint64) error {
+// concurrent request has already replaced it.
+func (c *Client) renewAuthentication(ctx context.Context, expected uint64) error {
if err := ctx.Err(); err != nil {
return err
}
c.authMu.Lock()
- if expected != nil && c.authGeneration != *expected {
+ if c.authGeneration != expected {
c.authMu.Unlock()
return nil
}
@@ -477,11 +488,7 @@ func (c *Client) renewAuthentication(ctx context.Context, expected *uint64) erro
loginCtx, cancelLogin := context.WithTimeout(context.WithoutCancel(ctx), c.sessionRenewalTimeout())
go func() {
defer cancelLogin()
- password := c.config.Password
- err := c.authenticateJSON(loginCtx, username, password)
- if shouldFallbackToForm(err) {
- err = c.authenticateForm(loginCtx, username, password)
- }
+ err := c.authenticatePassword(loginCtx, username)
c.authMu.Lock()
pending.err = err
c.authInFlight = nil
@@ -627,7 +634,7 @@ func (c *Client) requestWithRetryUsingClient(ctx context.Context, method, path s
// Re-authenticate if needed
auth, generation := c.authSnapshot()
if c.config.Password != "" && auth.tokenName == "" && time.Now().After(auth.expiresAt) {
- if err := c.renewAuthentication(ctx, &generation); err != nil {
+ if err := c.renewAuthentication(ctx, generation); err != nil {
return nil, fmt.Errorf("re-authentication failed: %w", err)
}
auth, generation = c.authSnapshot()
@@ -689,7 +696,7 @@ func (c *Client) requestWithRetryUsingClient(ctx context.Context, method, path s
// Password sessions can be invalidated server-side before local expiry.
// Retry once after forcing re-authentication on 401.
if resp.StatusCode == http.StatusUnauthorized && !retriedAfter401 && c.config.Password != "" && auth.tokenName == "" {
- if err := c.renewAuthentication(ctx, &generation); err != nil {
+ if err := c.renewAuthentication(ctx, generation); err != nil {
return nil, fmt.Errorf("re-authentication failed after 401: %w", err)
}
return c.requestWithRetryUsingClient(ctx, method, path, data, true, httpClient)
diff --git a/pkg/proxmox/client_construction_context_test.go b/pkg/proxmox/client_construction_context_test.go
index f8e95ececf..5ea2b7f1cf 100644
--- a/pkg/proxmox/client_construction_context_test.go
+++ b/pkg/proxmox/client_construction_context_test.go
@@ -16,7 +16,10 @@ import (
"time"
)
-func testClientConstructionHonoursContext(t *testing.T) {
+func testClientConstructionHonoursContext(t *testing.T, endings ...string) {
+ if len(endings) == 0 {
+ endings = []string{"cancel", "deadline"}
+ }
for _, mode := range []string{"password", "token"} {
t.Run("already-ended/"+mode, func(t *testing.T) {
var requests atomic.Int32
@@ -45,7 +48,7 @@ func testClientConstructionHonoursContext(t *testing.T) {
})
}
for _, encoding := range []string{"json", "form"} {
- for _, end := range []string{"cancel", "deadline"} {
+ for _, end := range endings {
t.Run(encoding+"/"+end, func(t *testing.T) {
var logins, reads atomic.Int32
var blocking atomic.Bool
@@ -84,6 +87,11 @@ func testClientConstructionHonoursContext(t *testing.T) {
defer releaseOnce.Do(func() { close(release) })
pin := sha256.Sum256(server.Certificate().Raw)
cfg := ClientConfig{Host: server.URL, User: "fixture@pam", Password: "fixture-password", VerifySSL: true, Fingerprint: hex.EncodeToString(pin[:]), Timeout: 8 * time.Second}
+ resultBound := time.Second
+ if end == "client-timeout" {
+ cfg.Timeout = time.Second
+ resultBound = 5 * time.Second
+ }
ctx, cancel := context.WithCancel(context.Background())
if end == "deadline" {
cancel()
@@ -105,18 +113,25 @@ func testClientConstructionHonoursContext(t *testing.T) {
}
if end == "cancel" {
cancel()
- } else {
+ } else if end == "deadline" {
<-ctx.Done()
}
var got result
select {
case got = <-done:
- case <-time.After(time.Second):
+ case <-time.After(resultBound):
t.Error("constructor exceeded ended caller budget")
releaseOnce.Do(func() { close(release) })
got = <-done
}
- if got.client != nil || !errors.Is(got.err, ctx.Err()) {
+ wantErr := ctx.Err()
+ if end == "client-timeout" {
+ wantErr = context.DeadlineExceeded
+ if ctx.Err() != nil {
+ t.Fatal("client timeout ended the independent caller")
+ }
+ }
+ if got.client != nil || !errors.Is(got.err, wantErr) {
t.Errorf("cancelled construction returned session or lost context: client=%t error=%v", got.client != nil, got.err)
}
select {
diff --git a/pkg/proxmox/host_diagnostics_regression_test.go b/pkg/proxmox/host_diagnostics_regression_test.go
index 2de7bebec9..2834a3f914 100644
--- a/pkg/proxmox/host_diagnostics_regression_test.go
+++ b/pkg/proxmox/host_diagnostics_regression_test.go
@@ -163,6 +163,10 @@ func TestClientConstructionHonoursContext(t *testing.T) {
testClientConstructionHonoursContext(t)
}
+func TestClientConstructionHonoursClientTimeout(t *testing.T) {
+ testClientConstructionHonoursContext(t, "client-timeout")
+}
+
func TestClusterCancelledCallerStartsNoSession(t *testing.T) {
testClusterCancelledCallerStartsNoSession(t)
}