Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Recover declared Proxmox members through password TLS admission

pulse-triage[bot]0c130694cc0a
diff --git a/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md b/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md
index ee19fa94e5..cb0217a281 100644
--- a/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md
+++ b/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md
@@ -1293,3 +1293,10 @@ as verification progresses:
1. `pending` means not yet confirmed end to end.
2. `blocked` means the gate is actively failing or cannot yet be exercised.
3. `passed` means both automated proof and the manual scenario are clear.
+
+### Proxmox declared-member first-use session admission
+
+- `TestClusterFirstUseSessionAdmission`: actual TLS, password JSON/form and token startup/recovery/selection; credential-free capture, pinned retry, full polling client and ordinary member read.
+- `TestClusterFirstUseSessionRefusalKeepsTrust`, `TestClusterPasswordEstablishedPinsAreNotFirstUse`, `TestClusterFirstUsePasswordCaptureCancellation`: rejected/malformed login does not become healthy or replay; primary/saved pins and ending callers cannot gain first-use authority.
+- `TestClusterFirstUsePasswordEndpointProjection`: real first-use password construction, failed retained session and separate normal public-read recovery feed the existing cluster-member consuming surface; native Online and saved identities/trust stay unchanged.
+- Exact-source race/vet/build and desktop/narrow consuming pixels are source qualification, not installed/native acceptance or release availability.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 465eadfb5e..863ef858ce 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -17,6 +17,30 @@
## Purpose
+### First-use cluster trust includes password construction
+
+Declared, previously untrusted Proxmox members use the existing credential-free
+TLS capture and pinned retry whether the inherited primary pin fails during
+password login or during the token-authenticated nodes probe. Startup, ordinary
+unhealthy recovery and lazy first selection share this path. Capture still
+requires typed local pin-mismatch evidence and cannot replace a primary, saved
+member or concurrent winner pin; it never sends an HTTP request or credentials.
+The pinned retry is not repeated to obtain a nodes count. Failed or cancelled
+authentication installs no polling session; a separately accepted full-timeout
+client is required before recovery becomes healthy. Refusal retains the
+established runtime pin for a distinct later ordinary recovery. Saved trust,
+CA/redirect policy, login encodings, cadence and guest/backup fences do not change.
+
+`TestClusterFirstUseSessionAdmission` covers JSON password, existing form
+fallback and token modes in all three phases, then ordinary member reads.
+The adjacent refusal, established-pin and cancellation controls exercise actual
+TLS sockets. `TestClusterFirstUsePasswordEndpointProjection` carries first-use
+probe/full-session refusal and later public-read recovery into the monitor's
+cluster-member projection without changing native Online state, identity or
+saved fingerprints. This is source/consumer qualification, not installed
+cluster recovery or attribution of an existing user report.
+
+
### Proxmox password success admits a complete session, not just HTTP 200
PVE and PBS consume the entire existing byte-bounded login body before
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index ef8b4c34c7..9715f43a8d 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -18,6 +18,30 @@
## Purpose
+### First-use cluster trust includes password construction
+
+Declared, previously untrusted Proxmox members use the existing credential-free
+TLS capture and pinned retry whether the inherited primary pin fails during
+password login or during the token-authenticated nodes probe. Startup, ordinary
+unhealthy recovery and lazy first selection share this path. Capture still
+requires typed local pin-mismatch evidence and cannot replace a primary, saved
+member or concurrent winner pin; it never sends an HTTP request or credentials.
+The pinned retry is not repeated to obtain a nodes count. Failed or cancelled
+authentication installs no polling session; a separately accepted full-timeout
+client is required before recovery becomes healthy. Refusal retains the
+established runtime pin for a distinct later ordinary recovery. Saved trust,
+CA/redirect policy, login encodings, cadence and guest/backup fences do not change.
+
+`TestClusterFirstUseSessionAdmission` covers JSON password, existing form
+fallback and token modes in all three phases, then ordinary member reads.
+The adjacent refusal, established-pin and cancellation controls exercise actual
+TLS sockets. `TestClusterFirstUsePasswordEndpointProjection` carries first-use
+probe/full-session refusal and later public-read recovery into the monitor's
+cluster-member projection without changing native Online state, identity or
+saved fingerprints. This is source/consumer qualification, not installed
+cluster recovery or attribution of an existing user report.
+
+
### Proxmox password success admits a complete session, not just HTTP 200
PVE and PBS consume the entire existing byte-bounded login body before
diff --git a/internal/monitoring/monitor_pve_first_use_session_test.go b/internal/monitoring/monitor_pve_first_use_session_test.go
new file mode 100644
index 0000000000..a49b74ea41
--- /dev/null
+++ b/internal/monitoring/monitor_pve_first_use_session_test.go
@@ -0,0 +1,131 @@
+package monitoring
+
+import (
+ "context"
+ "crypto/ed25519"
+ "crypto/rand"
+ "crypto/sha256"
+ "crypto/tls"
+ "crypto/x509"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "math/big"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+// Drive the password-construction TLS mismatch through the real cluster and
+// monitor projection. The short pinned probe is accepted, but the first full
+// polling session is refused; ordinary later recovery must precede removing
+// the consumer's Unreachable state. No guest, backup or native host is used.
+func TestClusterFirstUsePasswordEndpointProjection(t *testing.T) {
+ var acceptMember atomic.Bool
+ var memberLogins atomic.Int32
+ endpoint := func(member bool) *httptest.Server {
+ pub, key, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ template := &x509.Certificate{SerialNumber: big.NewInt(1), NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour)}
+ der, err := x509.CreateCertificate(rand.Reader, template, template, pub, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/api2/json/access/ticket":
+ if member && memberLogins.Add(1) > 1 && !acceptMember.Load() {
+ w.WriteHeader(http.StatusForbidden)
+ fmt.Fprint(w, `{"message":"fixture full polling session refused"}`)
+ return
+ }
+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket","CSRFPreventionToken":"fixture-csrf"}}`)
+ case "/api2/json/nodes":
+ if !strings.Contains(r.Header.Get("Cookie"), "PVEAuthCookie=fixture-ticket") {
+ t.Error("node read lost password session")
+ }
+ fmt.Fprint(w, `{"data":[{"node":"primary","status":"online","cpu":0.125},{"node":"member","status":"online","cpu":0.25}]}`)
+ default:
+ t.Errorf("outside ordinary cluster reads: %s", r.URL.Path)
+ http.NotFound(w, r)
+ }
+ }))
+ s.TLS = &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}}
+ s.StartTLS()
+ t.Cleanup(s.Close)
+ return s
+ }
+ primary, member := endpoint(false), endpoint(true)
+ sum := sha256.Sum256(primary.Certificate().Raw)
+ primaryPin := hex.EncodeToString(sum[:])
+ cc := proxmox.NewClusterClient("first-use-cluster", proxmox.ClientConfig{Host: primary.URL, User: "fixture@pam", Password: "fixture-password", VerifySSL: true, Fingerprint: primaryPin, Timeout: 2 * time.Second}, []string{primary.URL, member.URL}, nil)
+ if !cc.GetHealthStatus()[primary.URL] || cc.GetHealthStatus()[member.URL] || memberLogins.Load() != 2 {
+ t.Fatalf("first-use probe/full-session admission was not distinguished: member logins=%d", memberLogins.Load())
+ }
+ inst := config.PVEInstance{Name: "first-use-cluster", Host: primary.URL, IsCluster: true, ClusterName: "Fixture cluster", VerifySSL: true, Fingerprint: primaryPin, ClusterEndpoints: []config.ClusterEndpoint{
+ {NodeName: "primary", NodeIdentity: "primary-identity", Host: primary.URL, Fingerprint: primaryPin, DisplayName: "Trusted primary"},
+ {NodeName: "member", NodeIdentity: "member-identity", Host: member.URL, DisplayName: "New member"},
+ }}
+ m := &Monitor{config: &config.Config{PVEInstances: []config.PVEInstance{inst}}}
+ project := func(phase string) {
+ nodes, err := cc.GetNodes(context.Background())
+ if err != nil || len(nodes) != 2 {
+ t.Fatalf("trusted ordinary read failed: %+v %v", nodes, err)
+ }
+ model := make([]models.Node, len(nodes))
+ for i, n := range nodes {
+ model[i] = models.Node{Name: n.Node, Status: n.Status}
+ }
+ m.updateClusterEndpointStatus(inst.Name, &inst, cc, model)
+ for i, ep := range inst.ClusterEndpoints {
+ if !ep.Online || ep.NodeIdentity != []string{"primary-identity", "member-identity"}[i] {
+ t.Fatal("Pulse reachability changed native status or identity")
+ }
+ }
+ if inst.Fingerprint != primaryPin || inst.ClusterEndpoints[0].Fingerprint != primaryPin || inst.ClusterEndpoints[1].Fingerprint != "" {
+ t.Fatal("runtime first-use trust changed saved configuration")
+ }
+ b, err := json.Marshal(struct {
+ Phase string `json:"phase"`
+ Endpoints []config.ClusterEndpoint `json:"endpoints"`
+ }{phase, inst.ClusterEndpoints})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("FIRST_USE_ENDPOINT_PROJECTION_JSON=%s", b)
+ }
+ project("full-session-refused")
+ if ep := inst.ClusterEndpoints[1]; ep.PulseReachable == nil || *ep.PulseReachable || ep.PulseError != "Authentication failed - check API token or credentials" {
+ t.Fatalf("consumer lost failed polling-session reason: %+v", ep)
+ }
+ acceptMember.Store(true)
+ // Preserve the real ten-second recovery interval and use public reads to
+ // initiate the ordinary background recovery, rather than calling internals.
+ time.Sleep(11 * time.Second)
+ if _, err := cc.GetNodes(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ deadline := time.Now().Add(5 * time.Second)
+ for !cc.GetHealthStatus()[member.URL] {
+ if time.Now().After(deadline) {
+ t.Fatal("later normal recovery did not accept member")
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+ if memberLogins.Load() != 4 {
+ t.Fatalf("separate accepted probe/full session logins=%d", memberLogins.Load())
+ }
+ project("member-recovered")
+ if ep := inst.ClusterEndpoints[1]; ep.PulseReachable == nil || !*ep.PulseReachable || ep.PulseError != "" {
+ t.Fatalf("accepted polling session did not clear consumer error: %+v", ep)
+ }
+}
diff --git a/pkg/proxmox/cluster_client.go b/pkg/proxmox/cluster_client.go
index 03a26ae803..4b3b2c5c1a 100644
--- a/pkg/proxmox/cluster_client.go
+++ b/pkg/proxmox/cluster_client.go
@@ -372,6 +372,30 @@ func (cc *ClusterClient) refreshTOFUFingerprintAndRetry(ctx context.Context, end
return retryClient, err, true
}
+// probeEndpoint applies the same first-use boundary whether TLS verification
+// fails during password construction or the subsequent token-authenticated
+// nodes read. Capture is still credential-free and only a declared, unknown
+// member can establish a pin; a saved/primary pin is never replaced.
+// The returned client is a disposable probe, not an accepted polling session.
+func (cc *ClusterClient) probeEndpoint(ctx context.Context, cfg ClientConfig) (*Client, []Node, error) {
+ client, err := NewClientWithContext(ctx, cfg)
+ var nodes []Node
+ if err == nil {
+ probeCtx, cancel := context.WithTimeout(ctx, cfg.Timeout)
+ nodes, err = client.GetNodes(probeCtx)
+ cancel()
+ }
+ if retryClient, retryErr, retried := cc.refreshTOFUFingerprintAndRetry(ctx, cfg.Host, cfg.Timeout, err); retried {
+ if client != nil {
+ client.httpClient.CloseIdleConnections()
+ }
+ // The pinned retry already performed its nodes read. Do not repeat it
+ // just to fill a diagnostic count or after a refused password login.
+ return retryClient, nil, retryErr
+ }
+ return client, nodes, err
+}
+
// initialHealthCheck performs a quick parallel health check on all endpoints
func (cc *ClusterClient) initialHealthCheck(ctx context.Context) {
// Skip initial health check if there's only one endpoint
@@ -399,32 +423,9 @@ func (cc *ClusterClient) initialHealthCheck(ctx context.Context) {
cfg.Fingerprint = cc.getEndpointFingerprint(ep)
cfg.Timeout = 5 * time.Second
- testClient, err := NewClientWithContext(ctx, cfg)
- if err != nil {
- cc.mu.Lock()
- cc.nodeHealth[ep] = false
- cc.lastError[ep] = sanitizeEndpointError(err.Error())
- cc.lastHealthCheck[ep] = time.Now()
- cc.mu.Unlock()
- log.Info().
- Str("cluster", cc.name).
- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(ep)).
- Err(err).
- Msg("Cluster endpoint marked unhealthy on initialization")
- return
- }
- defer testClient.httpClient.CloseIdleConnections()
-
- // Quick test with slightly longer timeout for initial check
- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- _, err = testClient.GetNodes(testCtx)
- cancel()
-
- if retryClient, retryErr, refreshed := cc.refreshTOFUFingerprintAndRetry(ctx, ep, 5*time.Second, err); refreshed {
- if retryClient != nil {
- defer retryClient.httpClient.CloseIdleConnections()
- }
- err = retryErr
+ testClient, _, err := cc.probeEndpoint(ctx, cfg)
+ if testClient != nil {
+ defer testClient.httpClient.CloseIdleConnections()
}
// Check if error is VM-specific (shouldn't affect health)
@@ -688,26 +689,20 @@ func (cc *ClusterClient) getHealthyClient(ctx context.Context) (*Client, error)
testCfg := cfg
testCfg.Timeout = 3 * time.Second
- testClient, err := NewClientWithContext(ctx, testCfg)
- if err != nil {
- if ctx.Err() != nil {
- return nil, ctx.Err()
- }
- // Mark as unhealthy
- cc.nodeHealth[selectedEndpoint] = false
- log.Debug().
- Str("cluster", cc.name).
- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(selectedEndpoint)).
- Err(err).
- Msg("Failed to create client for cluster endpoint")
- return nil, fmt.Errorf("failed to create client for %s: %w", securityutil.InfrastructureEndpointDiagnostic(selectedEndpoint), err)
+ // First-use capture owns cc.mu internally. Release selection's lock
+ // while probing, just as synchronous unhealthy recovery does above.
+ // Other readers can continue using an already accepted sibling.
+ var testClient *Client
+ var testNodes []Node
+ var testErr error
+ func() {
+ cc.mu.Unlock()
+ defer cc.mu.Lock()
+ testClient, testNodes, testErr = cc.probeEndpoint(ctx, testCfg)
+ }()
+ if testClient != nil {
+ defer testClient.httpClient.CloseIdleConnections()
}
- defer testClient.httpClient.CloseIdleConnections()
-
- // Connectivity test - 5 seconds to allow for TLS handshake (~3s typical)
- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- testNodes, testErr := testClient.GetNodes(testCtx)
- cancel()
if err := ctx.Err(); err != nil {
return nil, err
}
@@ -754,6 +749,8 @@ func (cc *ClusterClient) getHealthyClient(ctx context.Context) (*Client, error)
Int("nodes", len(testNodes)).
Msg("Cluster endpoint passed connectivity test")
+ // Use the first-use winner's pin for the full polling session too.
+ cfg.Fingerprint = cc.getEndpointFingerprintLocked(selectedEndpoint)
// Create the actual client with full timeout
newClient, err := NewClientWithContext(ctx, cfg)
if err != nil {
@@ -886,29 +883,9 @@ func (cc *ClusterClient) recoverUnhealthyNodes(ctx context.Context) {
cfg.Fingerprint = cc.getEndpointFingerprint(ep)
cfg.Timeout = 5 * time.Second
- testClient, err := NewClientWithContext(ctx, cfg)
- if err != nil {
- recordFailure(ep, err)
- log.Debug().
- Str("cluster", cc.name).
- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(ep)).
- Err(err).
- Msg("Failed to create client during recovery attempt")
- return
- }
- defer testClient.httpClient.CloseIdleConnections()
-
- // Try a simple API call
- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- _, err = testClient.GetNodes(testCtx)
- cancel()
-
- if retryClient, retryErr, refreshed := cc.refreshTOFUFingerprintAndRetry(ctx, ep, 5*time.Second, err); refreshed {
- if retryClient != nil {
- defer retryClient.httpClient.CloseIdleConnections()
- }
- err = retryErr
- cfg.Fingerprint = cc.getEndpointFingerprint(ep)
+ testClient, _, err := cc.probeEndpoint(ctx, cfg)
+ if testClient != nil {
+ defer testClient.httpClient.CloseIdleConnections()
}
if ctx.Err() != nil {
return
diff --git a/pkg/proxmox/cluster_client_api_test.go b/pkg/proxmox/cluster_client_api_test.go
index 02719163bb..8034b4275f 100644
--- a/pkg/proxmox/cluster_client_api_test.go
+++ b/pkg/proxmox/cluster_client_api_test.go
@@ -748,3 +748,9 @@ func TestClusterClientFirstUseCaptureCancellation(t *testing.T) {
})
}
}
+
+// Startup, ordinary recovery and first selection share the same declared-member
+// boundary for password and token authentication, not just post-construction I/O.
+func TestClusterFirstUseSessionAdmission(t *testing.T) {
+ testClusterFirstUseSessionAdmission(t)
+}
diff --git a/pkg/proxmox/cluster_client_first_use_session_test.go b/pkg/proxmox/cluster_client_first_use_session_test.go
new file mode 100644
index 0000000000..2712141cf3
--- /dev/null
+++ b/pkg/proxmox/cluster_client_first_use_session_test.go
@@ -0,0 +1,299 @@
+package proxmox
+
+import (
+ "context"
+ "crypto/tls"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+)
+
+type firstUseSessionFixture struct {
+ server *httptest.Server
+ cert atomic.Pointer[tls.Certificate]
+ handshakes, logins, nodes atomic.Int32
+ refuse atomic.Int32
+ gateAt atomic.Int32
+ entered, resume chan struct{}
+ form bool
+ token bool
+}
+
+func newFirstUseSessionFixture(t *testing.T, token, form bool) *firstUseSessionFixture {
+ t.Helper()
+ f := &firstUseSessionFixture{entered: make(chan struct{}), resume: make(chan struct{}), token: token, form: form}
+ f.cert.Store(clusterTrustCertificate(t))
+ f.server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/api2/json/access/ticket":
+ f.logins.Add(1)
+ if token {
+ t.Error("API token triggered password login")
+ }
+ if form && strings.Contains(r.Header.Get("Content-Type"), "application/json") {
+ w.WriteHeader(http.StatusUnsupportedMediaType)
+ return
+ }
+ var credentials map[string]string
+ if strings.Contains(r.Header.Get("Content-Type"), "application/json") {
+ if err := json.NewDecoder(r.Body).Decode(&credentials); err != nil {
+ t.Error(err)
+ }
+ } else {
+ if err := r.ParseForm(); err != nil {
+ t.Error(err)
+ }
+ credentials = map[string]string{"username": r.Form.Get("username"), "password": r.Form.Get("password")}
+ }
+ if credentials["username"] != "fixture@pam" || credentials["password"] != "fixture-password" {
+ t.Error("password authentication changed")
+ }
+ switch f.refuse.Load() {
+ case 401, 403:
+ w.WriteHeader(int(f.refuse.Load()))
+ fmt.Fprint(w, `{"message":"fixture login refused"}`)
+ return
+ case 200:
+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket"}}`)
+ return
+ }
+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket","CSRFPreventionToken":"fixture-csrf"}}`)
+ case "/api2/json/nodes":
+ f.nodes.Add(1)
+ if token {
+ if r.Header.Get("Authorization") != "PVEAPIToken=fixture@pve!pulse=fixture" {
+ t.Error("token authentication changed")
+ }
+ } else if r.Header.Get("Cookie") != "PVEAuthCookie=fixture-ticket" || r.Header.Get("CSRFPreventionToken") != "" {
+ t.Error("ordinary read lost the coherent fixture session")
+ }
+ fmt.Fprint(w, `{"data":[{"node":"member","status":"online","cpu":0.25}]}`)
+ default:
+ t.Errorf("unexpected request %s", r.URL.Path)
+ http.NotFound(w, r)
+ }
+ }))
+ f.server.Config.ErrorLog = log.New(io.Discard, "", 0)
+ f.server.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
+ n := f.handshakes.Add(1)
+ cert := f.cert.Load()
+ if n == f.gateAt.Load() {
+ close(f.entered)
+ <-f.resume
+ }
+ return &tls.Config{MinVersion: tls.VersionTLS12, Certificates: []tls.Certificate{*cert}}, nil
+ }}
+ f.server.StartTLS()
+ t.Cleanup(f.server.Close)
+ return f
+}
+
+func firstUseSessionConfig(primary string, pin string, token bool) ClientConfig {
+ cfg := ClientConfig{Host: primary, User: "fixture@pam", Password: "fixture-password", VerifySSL: true, Fingerprint: pin, Timeout: 2 * time.Second}
+ if token {
+ cfg.Password = ""
+ cfg.TokenName = "fixture@pve!pulse"
+ cfg.TokenValue = "fixture"
+ }
+ return cfg
+}
+
+func closeFirstUseClusterClients(t *testing.T, cc *ClusterClient) {
+ t.Helper()
+ t.Cleanup(func() {
+ cc.mu.RLock()
+ defer cc.mu.RUnlock()
+ for _, c := range cc.clients {
+ if c != nil {
+ c.httpClient.CloseIdleConnections()
+ }
+ }
+ })
+}
+
+func runFirstUsePhase(t *testing.T, phase string, cfg ClientConfig, f *firstUseSessionFixture, pins map[string]string) *ClusterClient {
+ t.Helper()
+ var cc *ClusterClient
+ if phase == "startup" {
+ primary := newFirstUseSessionFixture(t, f.token, f.form)
+ cfg.Host = primary.server.URL
+ cfg.Fingerprint = clusterTrustPin(primary.cert.Load())
+ cc = NewClusterClient("first-use", cfg, []string{primary.server.URL, f.server.URL}, pins)
+ } else {
+ cc = NewClusterClient("first-use", cfg, []string{f.server.URL}, pins)
+ if phase == "recovery" {
+ cc.nodeHealth[f.server.URL] = false
+ cc.lastError[f.server.URL] = "fixture unavailable"
+ cc.recoverUnhealthyNodes(context.Background())
+ } else {
+ if _, err := cc.GetNodes(context.Background()); err != nil {
+ t.Errorf("lazy first-use read: %v", err)
+ }
+ }
+ }
+ closeFirstUseClusterClients(t, cc)
+ return cc
+}
+
+func testClusterFirstUseSessionAdmission(t *testing.T) {
+ for _, phase := range []string{"startup", "recovery", "selection"} {
+ for _, mode := range []string{"password-json", "password-form", "api-token"} {
+ t.Run(phase+"/"+mode, func(t *testing.T) {
+ token, form := mode == "api-token", mode == "password-form"
+ f := newFirstUseSessionFixture(t, token, form)
+ primaryPin := clusterTrustPin(clusterTrustCertificate(t))
+ cfg := firstUseSessionConfig("https://primary.session.invalid", primaryPin, token)
+ saved := map[string]string{}
+ cc := runFirstUsePhase(t, phase, cfg, f, saved)
+ cc.mu.RLock()
+ client := cc.clients[f.server.URL]
+ healthy := cc.nodeHealth[f.server.URL]
+ cc.mu.RUnlock()
+ if !healthy || client == nil || cc.getEndpointFingerprint(f.server.URL) != clusterTrustPin(f.cert.Load()) {
+ t.Fatalf("declared new member has no usable pinned session: healthy=%t client=%t logins=%d nodes=%d", healthy, client != nil, f.logins.Load(), f.nodes.Load())
+ }
+ if len(saved) != 0 || cc.config.VerifySSL != true || cc.config.Fingerprint == cc.getEndpointFingerprint(f.server.URL) {
+ t.Fatal("first use changed saved trust or primary policy")
+ }
+ if client.config.Timeout != cfg.Timeout {
+ t.Fatal("retained polling timeout changed")
+ }
+ loginWant := int32(2)
+ if form {
+ loginWant = 4
+ }
+ if token {
+ loginWant = 0
+ }
+ nodesWant := int32(1)
+ if phase == "selection" {
+ nodesWant++
+ }
+ if f.logins.Load() != loginWant || f.nodes.Load() != nodesWant || f.handshakes.Load() != 4-int32(boolInt(token && phase != "selection")) {
+ t.Fatalf("probe/capture/full admission counts: login=%d nodes=%d handshakes=%d", f.logins.Load(), f.nodes.Load(), f.handshakes.Load())
+ }
+ // Use the actually retained full-timeout session through ordinary routing.
+ cc.mu.Lock()
+ for ep := range cc.nodeHealth {
+ if ep != f.server.URL {
+ cc.nodeHealth[ep] = false
+ cc.lastHealthCheck[ep] = time.Now()
+ }
+ }
+ cc.mu.Unlock()
+ nodes, err := cc.GetNodes(context.Background())
+ if err != nil || len(nodes) != 1 || nodes[0].CPU != 0.25 || f.logins.Load() != loginWant || f.nodes.Load() != nodesWant+1 {
+ t.Fatalf("ordinary member read failed or logged in again: %+v %v", nodes, err)
+ }
+ t.Logf("FIRST_USE_SESSION_FACTS phase=%s mode=%s saved-pins-unchanged=true primary-pin-preserved=true capture-http=0 retained-full-session=true nodes=%d logins=%d", phase, mode, f.nodes.Load(), f.logins.Load())
+ })
+ }
+ }
+}
+
+func boolInt(b bool) int {
+ if b {
+ return 1
+ }
+ return 0
+}
+
+func TestClusterFirstUseSessionRefusalKeepsTrust(t *testing.T) {
+ for _, failure := range []int32{401, 403, 200} {
+ t.Run(fmt.Sprint(failure), func(t *testing.T) {
+ f := newFirstUseSessionFixture(t, false, false)
+ f.refuse.Store(failure)
+ cfg := firstUseSessionConfig("https://primary.session.invalid", clusterTrustPin(clusterTrustCertificate(t)), false)
+ cc := runFirstUsePhase(t, "recovery", cfg, f, nil)
+ if cc.GetHealthStatus()[f.server.URL] || cc.clients[f.server.URL] != nil || f.logins.Load() != 1 || f.nodes.Load() != 0 {
+ t.Fatalf("refused first-use session became healthy or replayed: login=%d nodes=%d", f.logins.Load(), f.nodes.Load())
+ }
+ pin := cc.getEndpointFingerprint(f.server.URL)
+ if pin != clusterTrustPin(f.cert.Load()) || cc.lastError[f.server.URL] == "" {
+ t.Fatal("refusal lost its fixed pin or diagnostic")
+ }
+ // Eligibility changes only in the fixture; no native clock/fence reset.
+ f.refuse.Store(0)
+ delete(cc.lastHealthCheck, f.server.URL)
+ before := f.handshakes.Load()
+ cc.recoverUnhealthyNodes(context.Background())
+ if !cc.GetHealthStatus()[f.server.URL] || cc.clients[f.server.URL] == nil || cc.getEndpointFingerprint(f.server.URL) != pin || f.handshakes.Load() != before+2 || f.logins.Load() != 3 || f.nodes.Load() != 1 {
+ t.Fatal("separate ordinary recovery failed, recaptured trust or replayed the refusal")
+ }
+ })
+ }
+}
+
+func TestClusterPasswordEstablishedPinsAreNotFirstUse(t *testing.T) {
+ for _, phase := range []string{"startup", "recovery", "selection"} {
+ for _, role := range []string{"primary", "member"} {
+ t.Run(phase+"/"+role, func(t *testing.T) {
+ f := newFirstUseSessionFixture(t, false, false)
+ trusted := f.cert.Load()
+ pin := clusterTrustPin(trusted)
+ f.cert.Store(clusterTrustCertificate(t))
+ cfg := firstUseSessionConfig(f.server.URL, pin, false)
+ pins := map[string]string{}
+ if role == "member" {
+ cfg.Host = "https://primary.session.invalid"
+ pins[f.server.URL] = pin
+ }
+ cc := NewClusterClient("known-password", cfg, []string{f.server.URL}, pins)
+ closeFirstUseClusterClients(t, cc)
+ switch phase {
+ case "startup":
+ cc.endpoints = append(cc.endpoints, "http://127.0.0.1:1")
+ cc.initialHealthCheck(context.Background())
+ case "recovery":
+ cc.nodeHealth[f.server.URL] = false
+ cc.recoverUnhealthyNodes(context.Background())
+ case "selection":
+ _, _ = cc.GetNodes(context.Background())
+ }
+ if f.logins.Load() != 0 || f.nodes.Load() != 0 || f.handshakes.Load() != 1 || cc.getEndpointFingerprint(f.server.URL) != pin || cc.GetHealthStatus()[f.server.URL] {
+ t.Fatal("known leaf change captured new trust or sent a password")
+ }
+ f.cert.Store(trusted)
+ delete(cc.lastHealthCheck, f.server.URL)
+ cc.recoverUnhealthyNodes(context.Background())
+ if !cc.GetHealthStatus()[f.server.URL] || cc.getEndpointFingerprint(f.server.URL) != pin || f.logins.Load() != 2 || f.nodes.Load() != 1 {
+ t.Fatal("re-presenting trusted leaf did not recover")
+ }
+ })
+ }
+ }
+}
+
+func TestClusterFirstUsePasswordCaptureCancellation(t *testing.T) {
+ f := newFirstUseSessionFixture(t, false, false)
+ f.gateAt.Store(2)
+ cfg := firstUseSessionConfig("https://primary.session.invalid", clusterTrustPin(clusterTrustCertificate(t)), false)
+ cc := NewClusterClient("cancel-first-use", cfg, []string{f.server.URL}, nil)
+ closeFirstUseClusterClients(t, cc)
+ cc.nodeHealth[f.server.URL] = false
+ cc.lastError[f.server.URL] = "prior failure"
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan struct{})
+ go func() { cc.recoverUnhealthyNodes(ctx); close(done) }()
+ select {
+ case <-f.entered:
+ case <-time.After(5 * time.Second):
+ cancel()
+ close(f.resume)
+ <-done
+ t.Fatal("password construction mismatch did not reach credential-free capture")
+ }
+ cancel()
+ close(f.resume)
+ <-done
+ if len(cc.endpointFingerprints) != 0 || cc.lastError[f.server.URL] != "prior failure" || cc.GetHealthStatus()[f.server.URL] || f.logins.Load() != 0 || f.nodes.Load() != 0 {
+ t.Fatal("cancelled capture installed trust, changed failure or sent authentication")
+ }
+}